Mesos in production

Mesos is in production at several companies such as Apple, Twitter, and HubSpot and has even been used by start-ups such as Mattermark and Sigmoid. This broad appeal is a validation of Mesos' tremendous utility. Apple, for example, powers its consumer-facing, mission–critical, popular Siri application through a large Mesos cluster (allegedly spanning tens of thousands of nodes). One such case study (published on the Mesosphere website) is discussed here.

Case study on HubSpot

Following case study on HubSpot can be found here https://mesosphere.com/mesos-case-study-hubspot/. An excerpt from this link is given below:

HubSpot uses Apache Mesos to run a mixture of web services, long-running processes, and scheduled jobs that comprise their SaaS application. Mesos allows HubSpot to dynamically deploy services, which in turn reduces developer friction and time to deploy, increases reliability, achieves better resource utilization, and reduces hardware costs.

Mesos provides the core infrastructure to build a next-generation deployment system similar to what Heroku provides as a product. On top of Mesos, HubSpot built their own scheduler that is capable of executing both long-running services and scheduled jobs and is the interface through which the development team can view the state of their applications inside the cloud. Building a scheduler framework enables HubSpot to better understand the core concepts inside Mesos, be comfortable with failure modes, and customize user experience.

The cluster environment

Over 150 services run inside Mesos at HubSpot. HubSpot utilizes many hundreds of servers inside Amazon EC2, and the Mesos cluster comprises about 30% of these resources and is aggressively ramping up as more and more services are migrated to Mesos. As Mesos can easily handle large or small server footprints, hundreds of smaller servers are replaced with dozens of larger ones.

Benefits

Mesos provides numerous benefits to both the development team and the company. At HubSpot, developers own the operation of their applications. With Mesos, developers can deploy services faster and with less maintenance. Here are some of the other benefits:

• Developers get immediate access to cluster resources, whether it be to scale or introduce new services.
• Developers no longer need to understand the process of requisitioning hardware or servers, and it is easier to scale up the resource requirements inside Mesos than it is to recreate servers with more or less CPUs and memory.
• Hardware failures are more transparent to developers as services are automatically replaced when tasks are lost or they fail. In other words, developers are no longer paged because of a simple hardware failure.
• Scheduled tasks (cron jobs) are now exposed via a web interface and are not tied to a single server, which may fail at any time, taking the cron job with it.

Mesos also simplifies the technology stack required to requisition hardware and manage it from an operations perspective. HubSpot can standardize server footprints and simplify the base image upon which Mesos slaves are executed.

Lastly, resource utilization is improved, which directly corresponds with reducing costs. Services, which previously ran on overprovisioned hardware now use the exact amount of resources requested.

Additionally, the QA environment runs at 50% of its previous capacity as the HubSpot scheduler ensures that services are restarted when they fail. This means that it is no longer necessary to run multiple copies of services inside QA for high availability.

Challenges

A core challenge behind adoption is introducing a new deployment technology to a group of 100 engineers who are responsible for managing their applications on a daily basis. HubSpot mitigated this challenge by building a UI around Mesos and utilizing Mesos to make the deployment process as simple and rewarding as possible.

Looking ahead

HubSpot sees Mesos as a core technology behind future migrations into other datacenters. As both a virtualization and deployment technology, Mesos has proven to be a rewarding path forward. Additionally, HubSpot hopes to eventually leverage Mesos to dynamically scale out processes based on load, shrink and grow the cluster size relative to demand, and assist developers with resource estimation.

Tip

Detailed steps to download the code bundle are mentioned in the Preface of this book. Please have a look. The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Mastering-Mesos. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!