Introduction to frameworks
A Mesos framework sits between Mesos and the application and acts as a layer to manage task scheduling and execution. As its implementation is application-specific, the term is often used to refer to the application itself. Earlier, a Mesos framework could interact with the Mesos API using only the libmesos C++ library, due to which other language bindings were developed for Java, Scala, Python, and Go among others that leveraged libmesos heavily. Since v0.19.0, the changes made to the HTTP-based protocol enabled developers to develop frameworks using the language they wanted without having to rely on the C++ code. A framework consists of two components: a) Scheduler and b) Executor.
Scheduler is responsible for making decisions on the resource offers made to it and tracking the current state of the cluster. Communication with the Mesos master is handled by the SchedulerDriver module, which registers the framework with the master, launches tasks, and passes messages to other components.
The second component, Executor, is responsible, as its name suggests, for the execution of tasks on slave nodes. Communication with the slaves is handled by the ExecutorDriver module, which is also responsible for sending status updates to the scheduler.
The Mesos API, discussed later in this chapter, allows programmers to develop their own custom frameworks that can run on top of Mesos. Some other features of frameworks, such as authentication, authorization, and user management, will be discussed at length in Chapter 6, Mesos Frameworks.
Frameworks built on Mesos
A list of some of the services and frameworks built on Mesos is given here. This list is not exhaustive, and support for new frameworks is added almost every day. You can also refer to http://mesos.apache.org/documentation/latest/frameworks/ apart from the following list:
Long-running services
- Aurora: This is a service scheduler that runs on top of Mesos, enabling you to run long-running services that take advantage of the scalability, fault-tolerance, and resource isolation of Mesos.
- Marathon: This is a private PaaS built on Mesos. It automatically handles hardware or software failures and ensures that an app is "always on".
- Singularity: This is a scheduler (the HTTP API and web interface) for running Mesos tasks, such as long-running processes, one-off tasks, and scheduled jobs.
- SSSP: This is a simple web application that provides a "Megaupload" white label to store and share files in S3.
Big data processing
- Cray Chapel is a productive parallel programming language. The Chapel Mesos scheduler lets you run Chapel programs on Mesos.
- Dark is a Python clone of Spark, a MapReduce-like framework written in Python and running on Mesos.
- Exelixi is a distributed framework used to run genetic algorithms at scale.
- Hadoop Running Hadoop on Mesos distributes MapReduce jobs efficiently across an entire cluster.
- Hama is a distributed computing framework based on Bulk Synchronous Parallel computing techniques for massive scientific computations—for example, matrix, graph, and network algorithms.
- MPI is a message-passing system designed to function on a wide variety of parallel computers.
- Spark is a fast and general-purpose cluster computing system that makes parallel jobs easy to write.
- Storm is a distributed real-time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop does for batch processing.
Batch scheduling
- Chronos is a distributed job scheduler that supports complex job topologies. It can be used as a more fault-tolerant replacement for cron.
- Jenkins is a continuous integration server. The Mesos-Jenkins plugin allows it to dynamically launch workers on a Mesos cluster, depending on the workload.
- JobServer is a distributed job scheduler and processor that allows developers to build custom batch processing Tasklets using a point and click Web UI.
Data storage
- Cassandra is a performant and highly available distributed database. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data.
- Elasticsearch is a distributed search engine. Mesos makes it easy for it to run and scale.
