Mesos Architecture

The above figure shows the main components of Mesos. Mesos consists of a master daemon that manages agent daemons running on each cluster node, and Mesos frameworks that run tasks on these agents.

The master enables fine-grained sharing of resources (CPU, RAM, ...) across frameworks by making them resource offers. Each resource offer contains a list of <agent ID, resource1: amount1, resource2: amount2, ...> (NOTE: as keyword 'slave' is deprecated in favor of 'agent', driver-based frameworks will still receive offers with slave ID, whereas frameworks using the v1 HTTP API receive offers with agent ID). The master decides how many resources to offer to each framework according to a given organizational policy, such as fair sharing or strict priority. To support a diverse set of policies, the master employs a modular architecture that makes it easy to add new allocation modules via a plugin mechanism.

A framework running on top of Mesos consists of two components: a scheduler that registers with the master to be offered resources, and an executor process that is launched on agent nodes to run the framework's tasks (see the App/Framework development guide for more details about framework schedulers and executors). While the master determines how many resources are offered to each framework, the frameworks' schedulers select which of the offered resources to use. When a framework accepts offered resources, it passes to Mesos a description of the tasks it wants to run on them. In turn, Mesos launches the tasks on the corresponding agents.

Example of resource offer

The figure below shows an example of how a framework gets scheduled to run a task.

Let's walk through the events in the figure.

1. Agent 1 reports to the master that it has 4 CPUs and 4 GB of memory free. The master then invokes the allocation policy module, which tells it that framework 1 should be offered all available resources.
2. The master sends a resource offer describing what is available on agent 1 to framework 1.
3. The framework's scheduler replies to the master with information about two tasks to run on the agent, using <2 CPUs, 1 GB RAM> for the first task, and <1 CPUs, 2 GB RAM> for the second task.
4. Finally, the master sends the tasks to the agent, which allocates appropriate resources to the framework's executor, which in turn launches the two tasks (depicted with dotted-line borders in the figure). Because 1 CPU and 1 GB of RAM are still unallocated, the allocation module may now offer them to framework 2.

In addition, this resource offer process repeats when tasks finish and new resources become free.

While the thin interface provided by Mesos allows it to scale and allows the frameworks to evolve independently, one question remains: how can the constraints of a framework be satisfied without Mesos knowing about these constraints? For example, how can a framework achieve data locality without Mesos knowing which nodes store the data required by the framework? Mesos answers these questions by simply giving frameworks the ability to reject offers. A framework will reject the offers that do not satisfy its constraints and accept the ones that do. In particular, we have found that a simple policy called delay scheduling, in which frameworks wait for a limited time to acquire nodes storing the input data, yields nearly optimal data locality.

You can also read much more about the Mesos architecture in this technical paper.

Video and Slides of Mesos Presentations

(Listed in reverse chronological order)

MesosCon North America 2018

Video playlist + Slides

Jolt: Running Distributed, Fault-Tolerant Tests at Scale using Mesos

Video Sunil Shah, Kyle Kelly, and Timmy Zhu Presented November 1, 2017 at Bay Area Mesos User Group Meetup

MesosCon Europe 2017

Video playlist + Slides

MesosCon North America 2017

Video playlist + Slides

MesosCon Asia 2017

Video playlist + Slides

MesosCon Asia 2016

Video playlist + Slides

MesosCon Europe 2016

Slides

MesosCon North America 2016

Video playlist + Slides

MesosCon Europe 2015

Video playlist + Slides

MesosCon North America 2015

Video playlist + Slides

Building and Deploying Applications to Apache Mesos

Slides Joe Stein Presented February 26, 2015 at DigitalOcean Community Meetup

MesosCon 2014

Video playlist

Datacenter Computing with Apache Mesos

Slides Paco Nathan Presented April 15, 2014 at Big Data DC Meetup

Apache Spark at Viadeo (Running on Mesos)

Video + Slides Eugen Cepoi Presented April 9, 2014 at Paris Hadoop User Group

Mesos, HubSpot, and Singularity

Video Tom Petr Presented April 3rd, 2014 at @TwitterOSS #conf

Building Distributed Frameworks on Mesos

Video Benjamin Hindman Presented March 25th, 2014 at Aurora and Mesos Frameworks Meetup

Introduction to Apache Aurora

Video Bill Farner Presented March 25th, 2014 at Aurora and Mesos Frameworks Meetup

Improving Resource Efficiency with Apache Mesos

Video Christina Delimitrou Presented April 3rd, 2014 at @TwitterOSS #conf

Apache Mesos as an SDK for Building Distributed Frameworks

Slides Paco Nathan Presented February 13th, 2014 at Strata

Run your Data Center like Google's with Apache Mesos

Video and Demo Abhishek Parolkar Presented November 14th, 2013 at Cloud Expo Asia 2013

Datacenter Management with Mesos

Video Benjamin Hindman Presented August 29th, 2013 at AMP Camp

Building a Framework on Mesos: A Case Study with Jenkins

Video Vinod Kone Presented July 25, 2013 at SF Mesos Meetup

Hadoop on Mesos

Video Brenden Matthews Presented July 25, 2013 at SF Mesos Meetup

Introduction to Apache Mesos

Slides Benjamin Hindman Presented August 20, 2013 at NYC Mesos Meetup

Chronos: A Distributed, Fault-Tolerant and Highly Available Job Orchestration Framework for Mesos

Slides Florian Leibert Presented August 20, 2013 at NYC Mesos Meetup

Airbnb Tech Talk

Video Benjamin Hindman Presented September 6, 2012 at Airbnb

Managing Twitter Clusters with Mesos

Video Benjamin Hindman Presented August 22, 2012 at AMP Camp

Mesos: A Platform for Fine-Grained Resource Sharing in Datacenters

Video Matei Zaharia Presented March 2011 at UC Berkeley

Mesos: Efficiently Sharing the Datacenter

Video Benjamin Hindman Presented November 8, 2010 at LinkedIn

Mesos: A Resource Management Platform for Hadoop and Big Data Clusters

Video Matei Zaharia Presented Summer 2010 at Yahoo

Apache Mesos - Paid Training

Automated Machine Learning Pipeline with Mesos

Video Karl Whitford Packt (November 2017)

Docker, Apache Mesos & DCOS: Run and manage cloud datacenter (Video)

Manuj Aggarwal Packt (January 2018)

Mesos Release and Support policy

The Mesos versioning and release policy gives operators and developers clear guidelines on:

• Making modifications to the existing APIs without affecting backward compatibility.
• How long a Mesos API will be supported.
• Upgrading a Mesos installation across release versions.

This document describes the release strategy for Mesos post 1.0.0 release.

Release Schedule

Mesos releases are time-based, though we do make limited adjustments to the release schedule to accommodate feature development. This gives users and developers a predictable cadence to consume and produce features, while ensuring that each release can include the developments that users are waiting for.

If a feature is not ready by the time a release is cut, that feature should be disabled. This means that features should be developed in such a way that they are opt-in by default and can be easily disabled (e.g., flag).

A new Mesos release is cut approximately every 3 months. The versioning scheme is SemVer. Typically, the minor release version is incremented by 1 (e.g., 1.1, 1.2, 1.3 etc) for every release, unless it is a major release.

Every (minor) release is a stable release and recommended for production use. This means a release candidate will go through rigorous testing (unit tests, integration tests, benchmark tests, cluster tests, scalability, etc.) before being officially released. In the rare case that a regular release is not deemed stable, a patch release will be released that will stabilize it.

At any given time, 3 releases are supported: the latest release and the two prior. Support means fixing of critical issues that affect the release. Once an issue is deemed critical, it will be fixed in only those affected releases that are still supported. This is called a patch release and increments the patch version by 1 (e.g., 1.2.1). Once a release reaches End Of Life (i.e., support period has ended), no more patch releases will be made for that release. Note that this is not related to backwards compatibility guarantees and deprecation periods (discussed later).

Which issues are considered critical?

• Security fixes
• Compatibility regressions
• Functional regressions
• Performance regressions
• Fixes for 3rd party integration (e.g., Docker remote API)

Whether an issue is considered critical or not is sometimes subjective. In some cases it is obvious and sometimes it is fuzzy. Users should work with committers to figure out the criticality of an issue and get agreement and commitment for support.

Patch releases are normally done once per month.

If a particular issue is affecting a user and the user cannot wait until the next scheduled patch release, they can request an off-schedule patch release for a specific supported version. This should be done by sending an email to the dev list.

Upgrades

All stable releases will be loosely compatible. Loose compatibility means:

• Master or agent can be upgraded to a new release version as long as they or the ecosystem components (scheduler, executor, zookeeper, service discovery layer, monitoring etc) do not depend on deprecated features (e.g., deprecated flags, deprecated metrics).
• There should be no unexpected effect on externally visible behavior that is not deprecated. See API compatibility section for what should be expected for Mesos APIs.

NOTE: The compatibility guarantees do not apply to modules yet. See Modules section below for details.

This means users should be able to upgrade (as long as they are not depending on deprecated / removed features) Mesos master or agent from a stable release version N directly to another stable release version M without having to go through intermediate release versions. For the purposes of upgrades, a stable release means the release with the latest patch version. For example, among 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.4.1 releases 1.2.1, 1.3.0 and 1.4.1 are considered stable and so a user should be able to upgrade from 1.2.1 directly to 1.4.1. Look at the API compatability section below for how frameworks can do seamless upgrades.

The deprecation period for any given feature will be 6 months. Having a set period allows Mesos developers to not indefinitely accrue technical debt and allows users time to plan for upgrades.

The detailed information about upgrading to a particular Mesos version would be posted here.

API versioning

The Mesos APIs (constituting Scheduler, Executor, Internal, Operator/Admin APIs) will have a version in the URL. The versioned URL will have a prefix of /api/vN where "N" is the version of the API. The "/api" prefix is chosen to distinguish API resources from Web UI paths.

Examples:

• http://localhost:5050/api/v1/scheduler : Scheduler HTTP API hosted by the master.
• http://localhost:5051/api/v1/executor : Executor HTTP API hosted by the agent.

A given Mesos installation might host multiple versions of the same API i.e., Scheduler API v1 and/or v2 etc.

API version vs Release version

• To keep things simple, the stable version of the API will correspond to the major release version of Mesos.
  • For example, v1 of the API will be supported by Mesos release versions 1.0.0, 1.4.0, 1.20.0 etc.
• vN version of the API might also be supported by release versions of N-1 series but the vN API is not considered stable until the last release version of N-1 series.
• For example, v2 of the API might be introduced in Mesos 1.12.0 release but it is only considered stable in Mesos 1.21.0 release if it is the last release of "1" series. Note that all Mesos 1.x.y versions will still support v1 of the API.
• The API version is only bumped if we need to make a backwards incompatible API change. We will strive to support a given API version for at least a year.
• The deprecation clock for vN-1 API will start as soon as we release "N.0.0" version of Mesos. We will strive to give enough time (e.g., 6 months) for frameworks/operators to upgrade to vN API before we stop supporting vN-1 API.

API Compatibility

The API compatibility is determined by the corresponding protobuf guarantees.

As an example, the following are considered "backwards compatible" changes for Scheduler API:

• Adding new types of Calls i.e., new types of HTTP requests to "/scheduler".
• Adding new optional fields to existing requests to "/scheduler".
• Adding new types of Events i.e., new types of chunks streamed on "/scheduler".
• Adding new header fields to chunked response streamed on "/scheduler".
• Adding new fields (or changing the order of fields) to chunks' body streamed on "/scheduler".
• Adding new API resources (e.g., "/foobar").

The following are considered backwards incompatible changes for Scheduler API:

• Adding new required fields to existing requests to "/scheduler".
• Renaming/removing fields from existing requests to "/scheduler".
• Renaming/removing fields from chunks streamed on "/scheduler".
• Renaming/removing existing Calls.

Implementation Details

Release branches

For regular releases, the work is done on the master branch. There are no feature branches but there will be release branches.

When it is time to cut a minor release, a new branch (e.g., 1.2.x) is created off the master branch. We chose 'x' instead of patch release number to disambiguate branch names from tag names. Then the first RC (-rc1) is tagged on the release branch. Subsequent RCs, in case the previous RCs fail testing, should be tagged on the release branch.

Patch releases are also based off the release branches. Typically the fix for an issue that is affecting supported releases lands on the master branch and is then backported to the release branch(es). In rare cases, the fix might directly go into a release branch without landing on master (e.g., fix / issue is not applicable to master).

Having a branch for each minor release reduces the amount of work a release manager needs to do when it is time to do a release. It is the responsibility of the committer of a fix to commit it to all the affecting release branches. This is important because the committer has more context about the issue / fix at the time of the commit than a release manager at the time of release. The release manager of a minor release will be responsible for all its patch releases as well. Just like the master branch, history rewrites are not allowed in the release branch (i.e., no git push --force).

API protobufs

Most APIs in Mesos accept protobuf messages with a corresponding JSON field mapping. To support multiple versions of the API, we decoupled the versioned protobufs backing the API from the "internal" protobufs used by the Mesos code.

For example, the protobufs for the v1 Scheduler API are located at:

include/mesos/v1/scheduler/scheduler.proto

package mesos.v1.scheduler;
option java_package = "org.apache.mesos.v1.scheduler";
option java_outer_classname = "Protos";
...

The corresponding internal protobufs for the Scheduler API are located at:

include/mesos/scheduler/scheduler.proto

package mesos.scheduler;
option java_package = "org.apache.mesos.scheduler";
option java_outer_classname = "Protos";
...

The users of the API send requests (and receive responses) based on the versioned protobufs. We implemented evolve/devolve converters that can convert protobufs from any supported version to the internal protobuf and vice versa.

Internally, message passing between various Mesos components would use the internal unversioned protobufs. When sending response (if any) back to the user of the API, the unversioned protobuf would be converted back to a versioned protobuf.

Building

Downloading Mesos

There are different ways you can get Mesos:

1. Download the latest stable release from Apache (Recommended)

$ wget https://downloads.apache.org/mesos/1.11.0/mesos-1.11.0.tar.gz
$ tar -zxf mesos-1.11.0.tar.gz

2. Clone the Mesos git repository (Advanced Users Only)

$ git clone https://gitbox.apache.org/repos/asf/mesos.git

NOTE: If you have problems running the above commands, you may need to first run through the System Requirements section below to install the wget, tar, and git utilities for your system.

System Requirements

Mesos runs on Linux (64 Bit) and Mac OS X (64 Bit). To build Mesos from source, GCC 4.8.1+ or Clang 3.5+ is required.

On Linux, a kernel version >= 2.6.28 is required at both build time and run time. For full support of process isolation under Linux a recent kernel >= 3.10 is required.

The Mesos agent also runs on Windows. To build Mesos from source, follow the instructions in the Windows section.

Make sure your hostname is resolvable via DNS or via /etc/hosts to allow full support of Docker's host-networking capabilities, needed for some of the Mesos tests. When in doubt, please validate that /etc/hosts contains your hostname.

Ubuntu 14.04

Following are the instructions for stock Ubuntu 14.04. If you are using a different OS, please install the packages accordingly.

# Update the packages.
$ sudo apt-get update

# Install a few utility tools.
$ sudo apt-get install -y tar wget git

# Install the latest OpenJDK.
$ sudo apt-get install -y openjdk-7-jdk

# Install autotools (Only necessary if building from git repository).
$ sudo apt-get install -y autoconf libtool

# Install other Mesos dependencies.
$ sudo apt-get -y install build-essential python-dev python-six python-virtualenv libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev

Ubuntu 16.04

Following are the instructions for stock Ubuntu 16.04. If you are using a different OS, please install the packages accordingly.

# Update the packages.
$ sudo apt-get update

# Install a few utility tools.
$ sudo apt-get install -y tar wget git

# Install the latest OpenJDK.
$ sudo apt-get install -y openjdk-8-jdk

# Install autotools (Only necessary if building from git repository).
$ sudo apt-get install -y autoconf libtool

# Install other Mesos dependencies.
$ sudo apt-get -y install build-essential python-dev python-six python-virtualenv libcurl4-nss-dev libsasl2-dev libsasl2-modules maven libapr1-dev libsvn-dev zlib1g-dev iputils-ping

Mac OS X 10.11 (El Capitan), macOS 10.12 (Sierra)

Following are the instructions for Mac OS X El Capitan. When building Mesos with the Apple-provided toolchain, the Command Line Tools from XCode >= 8.0 are required; XCode 8 requires Mac OS X 10.11.5 or newer.

# Install Python 3: https://www.python.org/downloads/

# Install Command Line Tools. The Command Line Tools from XCode >= 8.0 are required.
$ xcode-select --install

# Install Homebrew.
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

# Install Java.
$ brew install Caskroom/cask/java

# Install libraries.
$ brew install wget git autoconf automake libtool subversion maven xz

# Install Python dependencies.
$ sudo easy_install pip
$ pip install virtualenv

When compiling on macOS 10.12, the following is needed:

# There is an incompatiblity with the system installed svn and apr headers.
# We need the svn and apr headers from a brew installation of subversion.
# You may need to unlink the existing version of subversion installed via
# brew in order to configure correctly.
$ brew unlink subversion # (If already installed)
$ brew install subversion

# When configuring, the svn and apr headers from brew will be automatically
# detected, so no need to explicitly point to them.
# If the build fails due to compiler warnings, `--disable-werror` can be passed
# to configure to not treat warnings as errors.
$ ../configure

# Lastly, you may encounter the following error when the libprocess tests run:
$ ./libprocess-tests
Failed to obtain the IP address for '<hostname>'; the DNS service may not be able to resolve it: nodename nor servname provided, or not known

# If so, turn on 'Remote Login' within System Preferences > Sharing to resolve the issue.

NOTE: When upgrading from Yosemite to El Capitan, make sure to rerun xcode-select --install after the upgrade.

CentOS 6.6

Following are the instructions for stock CentOS 6.6. If you are using a different OS, please install the packages accordingly.

# Install a recent kernel for full support of process isolation.
$ sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
$ sudo rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
$ sudo yum --enablerepo=elrepo-kernel install -y kernel-lt

# Make the just installed kernel the one booted by default, and reboot.
$ sudo sed -i 's/default=1/default=0/g' /boot/grub/grub.conf
$ sudo reboot

# Install a few utility tools. This also forces an update of `nss`,
# which is necessary for the Java bindings to build properly.
$ sudo yum install -y tar wget git which nss

# 'Mesos > 0.21.0' requires a C++ compiler with full C++11 support,
# (e.g. GCC > 4.8) which is available via 'devtoolset-2'.
# Fetch the Scientific Linux CERN devtoolset repo file.
$ sudo wget -O /etc/yum.repos.d/slc6-devtoolset.repo http://linuxsoft.cern.ch/cern/devtoolset/slc6-devtoolset.repo

# Import the CERN GPG key.
$ sudo rpm --import http://linuxsoft.cern.ch/cern/centos/7/os/x86_64/RPM-GPG-KEY-cern

# Fetch the Apache Maven repo file.
$ sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo

# 'Mesos > 0.21.0' requires 'subversion > 1.8' devel package, which is
# not available in the default repositories.
# Create a WANdisco SVN repo file to install the correct version:
$ sudo bash -c 'cat > /etc/yum.repos.d/wandisco-svn.repo <<EOF
[WANdiscoSVN]
name=WANdisco SVN Repo 1.8
enabled=1
baseurl=http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/$basearch/
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF'

# Install essential development tools.
$ sudo yum groupinstall -y "Development Tools"

# Install 'devtoolset-2-toolchain' which includes GCC 4.8.2 and related packages.
# Installing 'devtoolset-3' might be a better choice since `perf` might
# conflict with the version of `elfutils` included in devtoolset-2.
$ sudo yum install -y devtoolset-2-toolchain

# Install other Mesos dependencies.
$ sudo yum install -y apache-maven python-devel python-six python-virtualenv java-1.7.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel

# Enter a shell with 'devtoolset-2' enabled.
$ scl enable devtoolset-2 bash
$ g++ --version  # Make sure you've got GCC > 4.8!

# Process isolation is using cgroups that are managed by 'cgconfig'.
# The 'cgconfig' service is not started by default on CentOS 6.6.
# Also the default configuration does not attach the 'perf_event' subsystem.
# To do this, add 'perf_event = /cgroup/perf_event;' to the entries in '/etc/cgconfig.conf'.
$ sudo yum install -y libcgroup
$ sudo service cgconfig start

CentOS 7.1

Following are the instructions for stock CentOS 7.1. If you are using a different OS, please install the packages accordingly.

# Install a few utility tools
$ sudo yum install -y tar wget git

# Fetch the Apache Maven repo file.
$ sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo

# Install the EPEL repo so that we can pull in 'libserf-1' as part of our
# subversion install below.
$ sudo yum install -y epel-release

# 'Mesos > 0.21.0' requires 'subversion > 1.8' devel package,
# which is not available in the default repositories.
# Create a WANdisco SVN repo file to install the correct version:
$ sudo bash -c 'cat > /etc/yum.repos.d/wandisco-svn.repo <<EOF
[WANdiscoSVN]
name=WANdisco SVN Repo 1.9
enabled=1
baseurl=http://opensource.wandisco.com/centos/7/svn-1.9/RPMS/\$basearch/
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF'

# Parts of Mesos require systemd in order to operate. However, Mesos
# only supports versions of systemd that contain the 'Delegate' flag.
# This flag was first introduced in 'systemd version 218', which is
# lower than the default version installed by centos. Luckily, centos
# 7.1 has a patched 'systemd < 218' that contains the 'Delegate' flag.
# Explicity update systemd to this patched version.
$ sudo yum update systemd

# Install essential development tools.
$ sudo yum groupinstall -y "Development Tools"

# Install other Mesos dependencies.
$ sudo yum install -y apache-maven python-devel python-six python-virtualenv java-1.8.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel

Windows

Follow the instructions in the Windows section.

Building Mesos (Posix)

# Change working directory.
$ cd mesos

# Bootstrap (Only required if building from git repository).
$ ./bootstrap

# Configure and build.
$ mkdir build
$ cd build
$ ../configure
$ make

In order to speed up the build and reduce verbosity of the logs, you can append -j <number of cores> V=0 to make.

# Run test suite.
$ make check

# Install (Optional).
$ make install

Examples

Mesos comes bundled with example frameworks written in C++, Java and Python. The framework binaries will only be available after running make check, as described in the Building Mesos section above.

# Change into build directory.
$ cd build

# Start Mesos master (ensure work directory exists and has proper permissions).
$ ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos

# Start Mesos agent (ensure work directory exists and has proper permissions).
$ ./bin/mesos-agent.sh --master=127.0.0.1:5050 --work_dir=/var/lib/mesos

# Visit the Mesos web page.
$ http://127.0.0.1:5050

# Run C++ framework (exits after successfully running some tasks).
$ ./src/test-framework --master=127.0.0.1:5050

# Run Java framework (exits after successfully running some tasks).
$ ./src/examples/java/test-framework 127.0.0.1:5050

# Run Python framework (exits after successfully running some tasks).
$ ./src/examples/python/test-framework 127.0.0.1:5050

Note: These examples assume you are running Mesos on your local machine. Following them will not allow you to access the Mesos web page in a production environment (e.g. on AWS). For that you will need to specify the actual IP of your host when launching the Mesos master and ensure your firewall settings allow access to port 5050 from the outside world.

Binary Packages

Downloading the Mesos RPM

Download and install the latest stable CentOS7 RPM binary from the Repository:

$ cat > /tmp/aventer.repo <<EOF
#aventer-mesos-el - packages by mesos from aventer
[aventer-rel]
name=AVENTER stable repository $releasever
baseurl=http://rpm.aventer.biz/CentOS/$releasever/$basearch/
enabled=1
gpgkey=https://www.aventer.biz/CentOS/support_aventer.asc
EOF

$ sudo mv /tmp/aventer.repo /etc/yum.repos.d/aventer.repo

$ sudo yum update

$ sudo yum install mesos

The above instructions show how to install the latest version of Mesos for RHEL 7. Substitute baseurl the with the appropriate URL for your operating system.

Start Mesos Master and Agent.

The RPM installation creates the directory /var/lib/mesos that can be used as a work directory.

Start the Mesos master with the following command:

$ mesos-master --work_dir=/var/lib/mesos

On a different terminal, start the Mesos agent, and associate it with the Mesos master started above:

$ mesos-agent --work_dir=/var/lib/mesos --master=127.0.0.1:5050

This is the simplest way to try out Mesos after downloading the RPM. For more complex and production setup instructions refer to the Administration section of the docs.

Mesos Runtime Configuration

The Mesos master and agent can take a variety of configuration options through command-line arguments or environment variables. A list of the available options can be seen by running mesos-master --help or mesos-agent --help. Each option can be set in two ways:

• By passing it to the binary using --option_name=value, either specifying the value directly, or specifying a file in which the value resides (--option_name=file://path/to/file). The path can be absolute or relative to the current working directory.

• By setting the environment variable MESOS_OPTION_NAME (the option name with a MESOS_ prefix added to it).

Configuration values are searched for first in the environment, then on the command-line.

Additionally, this documentation lists only a recent snapshot of the options in Mesos. A definitive source for which flags your version of Mesos supports can be found by running the binary with the flag --help, for example mesos-master --help.

Master and Agent Options

These are options common to both the Mesos master and agent.

See configuration/master-and-agent.md.

Master Options

See configuration/master.md.

Agent Options

See configuration/agent.md.

Libprocess Options

See configuration/libprocess.md.

Mesos Build Configuration

Autotools Options

If you have special compilation requirements, please refer to ./configure --help when configuring Mesos.

See configuration/autotools.md.

CMake Options

See configuration/cmake.md.

Install CMake 3.7+

Linux

Install the latest version of CMake from CMake.org. A self-extracting tarball is available to make this process painless.

Currently, few of the common Linux flavors package a sufficient CMake version. Ubuntu versions 12.04 and 14.04 package CMake 2; Ubuntu 16.04 packages CMake 3.5. If you already installed cmake from packages, you may remove it via: apt-get purge cmake.

The standard CentOS package is CMake 2, and unfortunately even the cmake3 package in EPEL is only CMake 3.6, you may remove them via: yum remove cmake cmake3.

Mac OS X

HomeBrew's CMake version is sufficient: brew install cmake.

Windows

Download and install the MSI from CMake.org.

NOTE: Windows needs CMake 3.8+, rather than 3.7+.

Quick Start

The most basic way to build with CMake, with no configuration, is fairly straightforward:

mkdir build
cd build
cmake ..
cmake --build .

The last step, cmake --build . can also take a --target command to build any particular target (e.g. mesos-tests, or tests to build mesos-tests, libprocess-tests, and stout-tests): cmake --build . --target tests. To send arbitrary flags to the native build system underneath (e.g. make), append the command with -- <flags to be passed>: cmake --build . -- -j4.

Also, cmake --build can be substituted by your build system of choice. For instance, the default CMake generator on Linux produces GNU Makefiles, so after configuring with cmake .., you can just run make tests in the build folder like usual. Similarly, if you configure with -G Ninja to use the Ninja generator, you can then run ninja tests to build the tests target with Ninja.

Installable build

This example will build Mesos and install it into a custom prefix:

mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/home/current_user/mesos
cmake --build . --target install

To additionally install mesos-tests executable and related test helpers (this can be used to run Mesos tests against the installed binaries), one can enable the MESOS_INSTALL_TESTS option.

To produce a set of binaries and libraries that will work after being copied/moved to a different location, use MESOS_FINAL_PREFIX.

The example below employs both MESOS_FINAL_PREFIX and MESOS_INSTALL_TESTS. On a build system:

mkdir build && cd build
cmake -DMESOS_FINAL_PREFIX=/opt/mesos -DCMAKE_INSTALL_PREFIX=/home/current_user/mesos -DMESOS_INSTALL_TESTS=ON
cmake --build . --target install
tar -czf mesos.tar.gz mesos -C /home/current_user

On a target system:

sudo tar -xf mesos.tar.gz -C /opt
# Run tests against Mesos installation
sudo /opt/mesos/bin/mesos-tests
# Start Mesos agent
sudo /opt/mesos/bin/mesos-agent --work-dir=/var/lib/mesos ...

Supported options

See configuration options.

Examples

See CMake By Example.

Documentation

The CMake documentation is written as a reference module. The most commonly used sections are:

• buildsystem overview
• commands
• properties
• variables

The wiki also has a set of useful variables.

Dependency graph

Like any build system, CMake has a dependency graph. The difference is that targets in CMake's dependency graph are much richer compared to other build systems. CMake targets have the notion of 'interfaces', where build properties are saved as part of the target, and these properties can be inherited transitively within the graph.

For example, say there is a library mylib, and anything which links it must include its headers, located in mylib/include. When building the library, some private headers must also be included, but not when linking to it. When compiling the executable myprogram, mylib's public headers must be included, but not its private headers. There is no manual step to add mylib/include to myprogram (and any other program which links to mylib), it is instead deduced from the public interface property of mylib. This is represented by the following code:

# A new library with a single source file (headers are found automatically).
add_library(mylib mylib.cpp)

# The folder of private headers, not exposed to consumers of `mylib`.
target_include_directories(mylib PRIVATE mylib/private)

# The folder of public headers, added to the compilation of any consumer.
target_include_directories(mylib PUBLIC mylib/include)

# A new exectuable with a single source file.
add_executable(myprogram main.cpp)

# The creation of the link dependency `myprogram` -> `mylib`.
target_link_libraries(myprogram mylib)

# There is no additional step to add `mylib/include` to `myprogram`.

This same notion applies to practically every build property: compile definitions via target_compile_definitions, include directories via target_include_directories, link libraries via target_link_libraries, compile options via target_compile_options, and compile features via target_compile_features.

All of these commands also take an optional argument of <INTERFACE|PUBLIC|PRIVATE>, which constrains their transitivity in the graph. That is, a PRIVATE include directory is recorded for the target, but not shared transitively to anything depending on the target, PUBLIC is used for both the target and dependencies on it, and INTERFACE is used only for dependencies.

Notably missing from this list are link directories. CMake explicitly prefers finding and using the absolute paths to libraries, obsoleting link directories.

Common mistakes

Booleans

CMake treats ON, OFF, TRUE, FALSE, 1, and 0 all as true/false booleans. Furthermore, variables of the form <target>-NOTFOUND are also treated as false (this is used for finding packages).

In Mesos, we prefer the boolean types TRUE and FALSE.

See if for more info.

Conditionals

For historical reasons, CMake conditionals such as if and elseif automatically interpolate variable names. It is therefore dangerous to interpolate them manually, because if ${FOO} evaluates to BAR, and BAR is another variable name, then if (${FOO}) becomes if (BAR), and BAR is then evaluated again by the if. Stick to if (FOO) to check the value of ${FOO}. Do not use if (${FOO}).

Also see the CMake policies CMP0012 and CMP0054.

Definitions

When using add_definitions() (which should be used rarely, as it is for "global" compile definitions), the flags must be prefixed with -D to be treated as preprocessor definitions. However, when using target_compile_definitions() (which should be preferred, as it is for specific targets), the flags do not need the prefix.

Style

In general, wrap at 80 lines, and use a two-space indent. When wrapping arguments, put the command on a separate line and arguments on subsequent lines:

target_link_libraries(
  program PRIVATE
  alpha
  beta
  gamma)

Otherwise keep it together:

target_link_libraries(program PUBLIC library)

Always keep the trailing parenthesis with the last argument.

Use a single space between conditionals and their open parenthesis, e.g. if (FOO), but not for commands, e.g. add_executable(program).

CAPITALIZE the declaration and use of custom functions and macros (e.g. EXTERNAL and PATCH_CMD), and do not capitalize the use of CMake built-in (including modules) functions and macros. CAPITALIZE variables.

CMake anti-patterns

Because CMake handles much more of the grunt work for you than other build systems, there are unfortunately a lot of CMake anti-patterns you should look out for when writing new CMake code. These are some common problems that should be avoided when writing new CMake code:

Superfluous use of add_dependencies

When you've linked library a to library b with target_link_libraries(a b), the CMake graph is already updated with the dependency information. It is redundant to use add_dependencies(a b) to (re)specify the dependency. In fact, this command should rarely be used.

The exceptions to this are:

1. Setting a dependency from an imported library to a target added via ExternalProject_Add.
2. Setting a dependency on Mesos modules since no explicit linking is done.
3. Setting a dependency between executables (e.g. the mesos-agent requiring the mesos-containerizer executable). In general, runtime dependencies need to be setup with add_dependency, but never link dependencies.

Use of link_libraries or link_directories

Neither of these commands should ever be used. The only appropriate command used to link libraries is target_link_libraries, which records the information in the CMake dependency graph. Furthermore, imported third-party libraries should have correct locations recorded in their respective targets, so the use of link_directories should never be necessary. The official documentation states:

Note that this command is rarely necessary. Library locations returned by find_package() and find_library() are absolute paths. Pass these absolute library file paths directly to the target_link_libraries() command. CMake will ensure the linker finds them.

The difference is that the former sets global (or directory level) side effects, and the latter sets specific target information stored in the graph.

Use of include_directories

This is similar to the above: the target_include_directories should always be preferred so that the include directory information remains localized to the appropriate targets.

Adding anything to endif ()

Old versions of CMake expected the style if (FOO) ... endif (FOO), where the endif contained the same expression as the if command. However, this is tortuously redundant, so leave the parentheses in endif () empty. This goes for other endings too, such as endforeach (), endwhile (), endmacro () and endfunction ().

Specifying header files superfluously

One of the distinct advantages of using CMake for C and C++ projects is that adding header files to the source list for a target is unnecessary. CMake is designed to parse the source files (.c, .cpp, etc.) and determine their required headers automatically. The exception to this is headers generated as part of the build (such as protobuf or the JNI headers).

Checking CMAKE_BUILD_TYPE

See the "Building debug or release configurations" example for more information. In short, not all generators respect the variable CMAKE_BUILD_TYPE at configuration time, and thus it must not be used in CMake logic. A usable alternative (where supported) is a generator expression such as $<$<CONFIG:Debug>:DEBUG_MODE>.

Remaining hacks

3RDPARTY_DEPENDENCIES

Until Mesos on Windows is stable, we keep some dependencies in an external repository, 3rdparty. When all dependencies are bundled with Mesos, this extra repository will no longer be necessary. Until then, the CMake variable 3RDPARTY_DEPENDENCIES points by default to this URL, but it can also point to the on-disk location of a local clone of the repo. With this option you can avoid pulling from GitHub for every clean build. Note that this must be an absolute path with forward slashes, e.g. -D3RDPARTY_DEPENDENCIES=C:/3rdparty, otherwise it will fail on Windows.

EXTERNAL

The CMake function EXTERNAL defines a few variables that make it easy for us to track the directory structure of a dependency. In particular, if our library's name is boost, we invoke:

EXTERNAL(boost ${BOOST_VERSION} ${CMAKE_CURRENT_BINARY_DIR})

Which will define the following variables as side-effects in the current scope:

• BOOST_TARGET (a target folder name to put dep in e.g., boost-1.53.0)
• BOOST_CMAKE_ROOT (where to have CMake put the uncompressed source, e.g., build/3rdparty/boost-1.53.0)
• BOOST_ROOT (where the code goes in various stages of build, e.g., build/.../boost-1.53.0/src, which might contain folders build-1.53.0-build, -lib, and so on, for each build step that dependency has)

The implementation is in 3rdparty/cmake/External.cmake.

This is not to be confused with the CMake module ExternalProject, from which we use ExternalProject_Add to download, extract, configure, and build our dependencies.

CMAKE_NOOP

This is a CMake variable we define in 3rdparty/CMakeLists.txt so that we can cancel steps of ExternalProject. ExternalProject's default behavior is to attempt to configure, build, and install a project using CMake. So when one of these steps must be skipped, we use set it to CMAKE_NOOP so that nothing is run instead.

CMAKE_FORWARD_ARGS

The CMAKE_FORWARD_ARGS variable defined in 3rdparty/CMakeLists.txt is sent as the CMAKE_ARGS argument to the ExternalProject_Add macro (along with any per-project arguments), and is used when the external project is configured as a CMake project. If either the CONFIGURE_COMMAND or BUILD_COMMAND arguments of ExternalProject_Add are used, then the CMAKE_ARGS argument will be ignored. This variable ensures that compilation configurations are properly propagated to third-party dependencies, such as compiler flags.

CMAKE_SSL_FORWARD_ARGS

The CMAKE_SSL_FORWARD_ARGS variable defined in 3rdparty/CMakeLists.txt is like CMAKE_FORWARD_ARGS, but only used for specific external projects that find and link against OpenSSL.

LIBRARY_LINKAGE

This variable is a shortcut used in 3rdparty/CMakeLists.txt. It is set to SHARED when BUILD_SHARED_LIBS is true, and otherwise it is set to STATIC. The SHARED and STATIC keywords are used to declare how a library should be built; however, if left out then the type is deduced automatically from BUILD_SHARED_LIBS.

MAKE_INCLUDE_DIR

This function works around a CMake issue with setting include directories of imported libraries built with ExternalProject_Add. We have to call this for each IMPORTED third-party dependency which has set INTERFACE_INCLUDE_DIRECTORIES, just to make CMake happy. An example is Glog:

MAKE_INCLUDE_DIR(glog)

GET_BYPRODUCTS

This function works around a CMake issue with the Ninja generator where it does not understand imported libraries, and instead needs BUILD_BYPRODUCTS explicitly set. This simply allows us to use ExternalProject_Add and Ninja. For Glog, it looks like this:

GET_BYPRODUCTS(glog)

Also see the CMake policy CMP0058.

PATCH_CMD

The CMake function PATCH_CMD generates a patch command given a patch file. If the path is not absolute, it's resolved to the current source directory. It stores the command in the variable name supplied. This is used to easily patch third-party dependencies. For Glog, it looks like this:

PATCH_CMD(GLOG_PATCH_CMD glog-${GLOG_VERSION}.patch)
ExternalProject_Add(
  ${GLOG_TARGET}
  ...
  PATCH_COMMAND     ${GLOG_PATCH_CMD})

The implementation is in 3rdparty/cmake/PatchCommand.cmake.

Windows patch.exe

While using patch on Linux is straightforward, doing the same on Windows takes a bit of work. PATH_CMD encapsulates this:

• Checks the cache variable PATCHEXE_PATH for patch.exe.
• Searches for patch.exe in its default locations.
• Copies patch.exe and a custom manifest to the temporary directory.
• Applies the manifest to avoid the UAC prompt.
• Uses the patched patch.exe.

As such, PATCH_CMD lets us apply patches as we do on Linux, without requiring an administrative prompt.

Note that on Windows, the patch file must have CRLF line endings. A file with LF line endings will cause the error: "Assertion failed, hunk, file patch.c, line 343". For this reason, it is required to checkout the Mesos repo with git config core.autocrlf true.

Windows

Mesos 1.0.0 introduced experimental support for Windows.

Building Mesos

System Requirements

1. Install the latest Visual Studio 2017: The "Community" edition is sufficient (and free of charge). During installation, choose the "Desktop development with C++" workload.

2. Install CMake 3.8.0 or later. During installation, choose to "Add CMake to the system PATH for all users".

3. Install GNU patch for Windows.

4. If building from source, install Git.

5. Make sure there are no spaces in your build directory. For example, C:/Program Files (x86)/mesos is an invalid build directory.

6. If developing Mesos, install Python 3 (not Python 2), in order to use our support scripts (e.g. to post and apply patches, or lint source code).

Build Instructions

Following are the instructions for Windows 10.

# Clone (or extract) Mesos.
git clone https://gitbox.apache.org/repos/asf/mesos.git
cd mesos

# Configure using CMake for an out-of-tree build.
mkdir build
cd build
cmake .. -G "Visual Studio 15 2017 Win64" -T "host=x64"

# Build Mesos.
# To build just the Mesos agent, add `--target mesos-agent`.
cmake --build .

# The Windows agent exposes new isolators that must be used as with
# the `--isolation` flag. To get started point the agent to a working
# master, using eiher an IP address or zookeeper information.
.\src\mesos-agent.exe --master=<master> --work_dir=<work folder> --launcher_dir=<repository>\build\src

Running Mesos

If you deploy the executables to another machine, you must also install the Microsoft Visual C++ Redistributable for Visual Studio 2017.

Known Limitations

The current implementation is known to have the following limitations:

• Only the agent should be run on Windows. The Mesos master can be launched, but only for testing as the master does not support high-availability setups on Windows.

• While Mesos supports NTFS long paths internally, tasks which do not support long paths must be run on agent whose --work_dir is a short path.

• The minimum versions of Windows supported are: Windows 10 Creators Update (AKA version 1703, build number 15063), and Windows Server, version 1709. It is likely that this will increase, due to evolving Windows container support and developer features which ease porting.

• The ability to create symlinks as a non-admin user requires Developer Mode to be enabled. Otherwise the agent will need to be run under an administrator.

Build Configuration Examples

Building with Ninja

Instead of using MSBuild, it is also possible to build Mesos on Windows using Ninja, which can result in significantly faster builds. To use Ninja, you need to download it and ensure ninja.exe is in your PATH.

• Download the Windows binary.
• Unzip it and place ninja.exe in your PATH.
• Open an "x64 Native Tools Command Prompt for VS 2017" to set your environment.
• In that command prompt, type powershell to use a better shell.
• Similar to above, configure CMake with cmake .. -G Ninja.
• Now you can use ninja to build the various targets.
• You may want to use ninja -v to make it verbose, as it's otherwise very quiet.

Note that with Ninja it is imperative to open the correct developer command prompt so that the 64-bit build tools are used, as Ninja does not otherwise know how to find them.

Building with Java

This enables more unit tests, but we do not yet officially produce mesos-master.

When building with Java on Windows, you must add the Maven build tool to your path. The JAVA_HOME environment variable must also be manually set. An installation of the Java SDK can be found form Oracle.

As of this writing, Java 9 is not yet supported, but Java 8 has been tested.

The Java build defaults to OFF because it is slow. To build the Java components on Windows, turn it ON:

mkdir build; cd build
$env:PATH += ";C:\...\apache-maven-3.3.9\bin\"
$env:JAVA_HOME = "C:\Program Files\Java\jdk1.8.0_144"
cmake .. -DENABLE_JAVA=ON -G "Visual Studio 15 2017 Win64" -T "host=x64"
cmake --build . --target mesos-java

Note that the mesos-java library does not have to be manually built; as libmesos will link it when Java is enabled.

Unfortunately, on Windows the FindJNI CMake module will populate JAVA_JVM_LIBRARY with the path to the static jvm.lib, but this variable must point to the shared library, jvm.dll, as it is loaded at runtime. Set it correctly like this:

$env:JAVA_JVM_LIBRARY = "C:\Program Files\Java\jdk1.8.0_144\jre\bin\server\jvm.dll"

The library may still fail to load at runtime with the following error:

"The specified module could not be found."

If this is the case, and the path to jvm.dll is verified to be correct, then the error message actually indicates that the dependencies of jvm.dll could not be found. On Windows, the DLL search path includes the environment variable PATH, so add the bin folder which contains server\jvm.dll to PATH:

$env:PATH += ";C:\Program Files\Java\jdk1.8.0_144\jre\bin"

Building with OpenSSL

When building with OpenSSL on Windows, you must build or install a distribution of OpenSSL for Windows. A commonly chosen distribution is Shining Light Productions' OpenSSL.

As of this writing, OpenSSL 1.1.x is supported.

Use -DENABLE_SSL=ON to build with OpenSSL.

Note that it will link to OpenSSL dynamically, so if the built executables are deployed elsewhere, that machine also needs OpenSSL installed.

Beware that the OpenSSL installation, nor Mesos itself, comes with a certificate bundle, and so it is likely that certificate verification will fail.

ClusterD Agent Options

Required Flags

Optional Flags

{
  "get_endpoints": [
    {
      "principals": { "values": ["a"] },
      "paths": { "values": ["/flags", "/monitor/statistics"] }
    }
  ]
}

{
    "capabilities": [
        {"type": "MULTI_ROLE"},
        {"type": "HIERARCHICAL_ROLE"},
        {"type": "RESERVATION_REFINEMENT"},
        {"type": "AGENT_OPERATION_FEEDBACK"},
        {"type": "RESOURCE_PROVIDER"},
        {"type": "AGENT_DRAINING"},
        {"type": "TASK_RESOURCE_LIMITS"}
    ]
}

{
  "capabilities": [
    "NET_RAW",
    "SYS_ADMIN"
  ]
}

  * /dev/console
  * /dev/tty0
  * /dev/tty1
  * /dev/pts/*
  * /dev/ptmx
  * /dev/net/tun
  * /dev/null
  * /dev/zero
  * /dev/full
  * /dev/tty
  * /dev/urandom
  * /dev/random

{
  "allowed_devices": [
    {
      "device": {
        "path": "/path/to/device"
      },
      "access": {
        "read": true,
        "write": false,
        "mknod": false
      }
    }
  ]
}

{
  "principal": "username",
  "secret": "secret"
}

{
  "mesos": [
    {
      "network_mode": "CNI",
      "network_name": "net1",
      "dns": {
        "nameservers": [ "8.8.8.8", "8.8.4.4" ]
      }
    }
  ],
  "docker": [
    {
      "network_mode": "BRIDGE",
      "dns": {
        "nameservers": [ "8.8.8.8", "8.8.4.4" ]
      }
    },
    {
      "network_mode": "USER",
      "network_name": "net2",
      "dns": {
        "nameservers": [ "8.8.8.8", "8.8.4.4" ]
      }
    }
  ]
}

{
  "type": "MESOS",
  "volumes": [
    {
      "host_path": ".private/tmp",
      "container_path": "/tmp",
      "mode": "RW"
    }
  ]
}

{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "xXxXxXxXxXx=",
      "email": "username@example.com"
    }
  }
}

{
  "PATH": "/bin:/usr/bin",
  "LD_LIBRARY_PATH": "/usr/local/lib"
}

{
  "credentials": [
    {
      "principal": "yoda",
      "secret": "usetheforce"
    }
  ]
}

{
  "image_disk_headroom": 0.1,
  "image_disk_watch_interval": {
    "nanoseconds": 3600000000000
  },
  "excluded_images": []
}

[
  {
    "name": "cpus",
    "type": "SCALAR",
    "scalar": {
      "value": 24
    }
  },
  {
    "name": "mem",
    "type": "SCALAR",
    "scalar": {
      "value": 24576
    }
  }
]

{
  "type": "org.mesos.apache.rp.local.storage",
  "name": "lvm"
}

{
  "type": "org.apache.mesos.csi.managed-plugin",
  "containers": [
    {
      "services": [
        "NODE_SERVICE"
      ],
      "command": {
        "value": " --endpoint=$CSI_ENDPOINT"
      },
      "resources": [
        {"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
        {"name": "mem", "type": "SCALAR", "scalar": {"value": 1024}}
      ]
    }
  ]
}

{
  "type": "org.apache.mesos.csi.unmanaged-plugin",
  "endpoints": [
    {
      "csi_service": "NODE_SERVICE",
      "endpoint": "/var/lib/unmanaged-plugin/csi.sock"
    }
  ],
  "target_path_root": "/mnt/unmanaged-plugin"
}

Network Isolator Flags

Available when configured with --with-network-isolator.

Seccomp Isolator flags

Available when configured with --enable-seccomp-isolator.

XFS Disk Isolator flags

Available when configured with --enable-xfs-disk-isolator.

Autotools Options

The most up-to-date options can be found with ./configure --help.

Autotools configure script options

Autotools configure script optional package flags

Environment variables which affect the Autotools configure script

Use these variables to override the choices made by configure or to help it to find libraries and programs with nonstandard names/locations.

CMake Options

The most up-to-date options can be found with cmake .. -LAH.

See more information in the CMake documentation.

Libprocess Options

The bundled libprocess library can be controlled with the following environment variables.

Master and Agent Options

These options can be supplied to both masters and agents.

{
  "disabled_endpoints" : {
    "paths" : [
      "/files/browse",
      "/metrics/snapshot"
    ]
  }
}

{
  "fault_domain":
    {
      "region":
        {
          "name": "aws-us-east-1"
        },
      "zone":
        {
          "name": "aws-us-east-1a"
        }
    }
}

{
  "libraries": [
    {
      "file": "/path/to/libfoo.so",
      "modules": [
        {
          "name": "org_apache_mesos_bar",
          "parameters": [
            {
              "key": "X",
              "value": "Y"
            }
          ]
        },
        {
          "name": "org_apache_mesos_baz"
        }
      ]
    },
    {
      "name": "qux",
      "modules": [
        {
          "name": "org_apache_mesos_norf"
        }
      ]
    }
  ]
}

Logging Options

These logging options can also be supplied to both masters and agents. For more about logging, see the logging documentation.

Master Options

Required Flags

zk://host1:port1,host2:port2,.../path
zk://username:password@host1:port1,host2:port2,.../path
file:///path/to/file (where file contains one of the above)

Optional Flags

{
  "register_frameworks": [
    {
      "principals": { "type": "ANY" },
      "roles": { "values": ["a"] }
    }
  ],
  "run_tasks": [
    {
      "principals": { "values": ["a", "b"] },
      "users": { "values": ["c"] }
    }
  ],
  "teardown_frameworks": [
    {
      "principals": { "values": ["a", "b"] },
      "framework_principals": { "values": ["c"] }
    }
  ],
  "set_quotas": [
    {
      "principals": { "values": ["a"] },
      "roles": { "values": ["a", "b"] }
    }
  ],
  "remove_quotas": [
    {
      "principals": { "values": ["a"] },
      "quota_principals": { "values": ["a"] }
    }
  ],
  "get_endpoints": [
    {
      "principals": { "values": ["a"] },
      "paths": { "values": ["/flags"] }
    }
  ]
}

{
  "credentials": [
    {
      "principal": "sherman",
      "secret": "kitesurf"
    }
  ]
}

{
  "limits": [
    {
      "principal": "foo",
      "qps": 55.5
    },
    {
      "principal": "bar"
    }
  ],
  "aggregate_default_qps": 33.3
}

Network Isolator Flags

Available when configured with --with-network-isolator.

Mesos Runtime Configuration

The Mesos master and agent can take a variety of configuration options through command-line arguments or environment variables. A list of the available options can be seen by running mesos-master --help or mesos-agent --help. Each option can be set in two ways:

• By passing it to the binary using --option_name=value, either specifying the value directly, or specifying a file in which the value resides (--option_name=file://path/to/file). The path can be absolute or relative to the current working directory.

• By setting the environment variable MESOS_OPTION_NAME (the option name with a MESOS_ prefix added to it).

Configuration values are searched for first in the environment, then on the command-line.

Additionally, this documentation lists only a recent snapshot of the options in Mesos. A definitive source for which flags your version of Mesos supports can be found by running the binary with the flag --help, for example mesos-master --help.

Master and Agent Options

These are options common to both the Mesos master and agent.

See configuration/master-and-agent.md.

Master Options

See configuration/master.md.

Agent Options

See configuration/agent.md.

Libprocess Options

See configuration/libprocess.md.

Mesos Build Configuration

Autotools Options

If you have special compilation requirements, please refer to ./configure --help when configuring Mesos.

See configuration/autotools.md.

CMake Options

See configuration/cmake.md.

Mesos High-Availability Mode

If the Mesos master is unavailable, existing tasks can continue to execute, but new resources cannot be allocated and new tasks cannot be launched. To reduce the chance of this situation occurring, Mesos has a high-availability mode that uses multiple Mesos masters: one active master (called the leader or leading master) and several backups in case it fails. The masters elect the leader, with Apache ZooKeeper both coordinating the election and handling leader detection by masters, agents, and scheduler drivers. More information regarding how leader election works is available on the Apache Zookeeper website.

This document describes how to configure Mesos to run in high-availability mode. For more information on developing highly available frameworks, see a companion document.

Note: This document assumes you know how to start, run, and work with ZooKeeper, whose client library is included in the standard Mesos build.

Usage

To put Mesos into high-availability mode:

1. Ensure that the ZooKeeper cluster is up and running.

2. Provide the znode path to all masters, agents, and framework schedulers as follows:

   • Start the mesos-master binaries using the --zk flag, e.g. --zk=zk://host1:port1,host2:port2,.../path

   • Start the mesos-agent binaries with --master=zk://host1:port1,host2:port2,.../path

   • Start any framework schedulers using the same zk path as in the last two steps. The SchedulerDriver must be constructed with this path, as shown in the Framework Development Guide.

From now on, the Mesos masters and agents all communicate with ZooKeeper to find out which master is the current leading master. This is in addition to the usual communication between the leading master and the agents.

In addition to ZooKeeper, one can get the location of the leading master by sending an HTTP request to /redirect endpoint on any master.

For HTTP endpoints that only work at the leading master, requests made to endpoints at a non-leading master will result in either a 307 Temporary Redirect (with the location of the leading master) or 503 Service Unavailable (if the master does not know who the current leader is).

Refer to the Scheduler API for how to deal with leadership changes.

Component Disconnection Handling

When a network partition disconnects a component (master, agent, or scheduler driver) from ZooKeeper, the component's Master Detector induces a timeout event. This notifies the component that it has no leading master. Depending on the component, the following happens. (Note that while a component is disconnected from ZooKeeper, a master may still be in communication with agents or schedulers and vice versa.)

• Agents disconnected from ZooKeeper no longer know which master is the leader. They ignore messages from masters to ensure they don't act on a non-leader's decisions. When an agent reconnects to ZooKeeper, ZooKeeper informs it of the current leader and the agent stops ignoring messages from the leader.

• Masters enter leaderless state irrespective of whether they are a leader or not before the disconnection.

  • If the leader was disconnected from ZooKeeper, it aborts its process. The user/developer/administrator can then start a new master instance which will try to reconnect to ZooKeeper.

    • Note that many production deployments of Mesos use a process supervisor (such as systemd or supervisord) that is configured to automatically restart the Mesos master if the process aborts unexpectedly.

  • Otherwise, the disconnected backup waits to reconnect with ZooKeeper and possibly get elected as the new leading master.

• Scheduler drivers disconnected from the leading master notify the scheduler about their disconnection from the leader.

When a network partition disconnects an agent from the leader:

• The agent fails health checks from the leader.

• The leader marks the agent as deactivated and sends its tasks to the LOST state. The Framework Development Guide describes these various task states.

• Deactivated agents may not reregister with the leader and are told to shut down upon any post-deactivation communication.

Monitoring

For monitoring the current number of masters in the cluster communicating with each other to form a quorum, see the monitoring guide's Replicated Log on registrar/log/ensemble_size. For creating alerts covering failures in leader election, have a look at the monitoring guide's Basic Alerts on master/elected.

Implementation Details

Mesos implements two levels of ZooKeeper leader election abstractions, one in src/zookeeper and the other in src/master (look for contender|detector.hpp|cpp).

• The lower level LeaderContender and LeaderDetector implement a generic ZooKeeper election algorithm loosely modeled after this recipe (sans herd effect handling due to the master group's small size, which is often 3).

• The higher level MasterContender and MasterDetector wrap around ZooKeeper's contender and detector abstractions as adapters to provide/interpret the ZooKeeper data.

• Each Mesos master simultaneously uses both a contender and a detector to try to elect themselves and detect who the current leader is. A separate detector is necessary because each master's WebUI redirects browser traffic to the current leader when that master is not elected. Other Mesos components (i.e., agents and scheduler drivers) use the detector to find the current leader and connect to it.

The notion of the group of leader candidates is implemented in Group. This abstraction handles reliable (through queues and retries of retryable errors under the covers) ZooKeeper group membership registration, cancellation, and monitoring. It watches for several ZooKeeper session events:

• Connection
• Reconnection
• Session Expiration
• ZNode creation, deletion, updates

We also explicitly timeout our sessions when disconnected from ZooKeeper for a specified amount of time. See --zk_session_timeout configuration option. This is because the ZooKeeper client libraries only notify of session expiration upon reconnection. These timeouts are of particular interest for network partitions.

The Mesos Replicated Log

Mesos provides a library that lets you create replicated fault-tolerant append-only logs; this library is known as the replicated log. The Mesos master uses this library to store cluster state in a replicated, durable way; the library is also available for use by frameworks to store replicated framework state or to implement the common "replicated state machine" pattern.

What is the replicated log?

The replicated log provides append-only storage of log entries; each log entry can contain arbitrary data. The log is replicated, which means that each log entry has multiple copies in the system. Replication provides both fault tolerance and high availability. In the following example, we use Apache Aurora, a fault tolerant scheduler (i.e., framework) running on top of Mesos, to show a typical replicated log setup.

As shown above, there are multiple Aurora instances running simultaneously (for high availability), with one elected as the leader. There is a log replica on each host running Aurora. Aurora can access the replicated log through a thin library containing the log API.

Typically, the leader is the only one that appends data to the log. Each log entry is replicated and sent to all replicas in the system. Replicas are strongly consistent. In other words, all replicas agree on the value of each log entry. Because the log is replicated, when Aurora decides to failover, it does not need to copy the log from a remote host.

Use Cases

The replicated log can be used to build a wide variety of distributed applications. For example, Aurora uses the replicated log to store all task states and job configurations. The Mesos master's registry also leverages the replicated log to store information about all agents in the cluster.

The replicated log is often used to allow applications to manage replicated state in a strongly consistent way. One way to do this is to store a state-mutating operation in each log entry and have all instances of the distributed application agree on the same initial state (e.g., empty state). The replicated log ensures that each application instance will observe the same sequence of log entries in the same order; as long as applying a state-mutating operation is deterministic, this ensures that all application instances will remain consistent with one another. If any instance of the application crashes, it can reconstruct the current version of the replicated state by starting at the initial state and re-applying all the logged mutations in order.

If the log grows too large, an application can write out a snapshot and then delete all the log entries that occurred before the snapshot. Using this approach, we will be exposing a distributed state abstraction in Mesos with replicated log as a backend.

Similarly, the replicated log can be used to build replicated state machines. In this scenario, each log entry contains a state machine command. Since replicas are strongly consistent, all servers will execute the same commands in the same order.

Implementation

The replicated log uses the Paxos consensus algorithm to ensure that all replicas agree on every log entry's value. It is similar to what's described in these slides. Readers who are familiar with Paxos can skip this section.

The above figure is an implementation overview. When a user wants to append data to the log, the system creates a log writer. The log writer internally creates a coordinator. The coordinator contacts all replicas and executes the Paxos algorithm to make sure all replicas agree about the appended data. The coordinator is sometimes referred to as the proposer.

Each replica keeps an array of log entries. The array index is the log position. Each log entry is composed of three components: the value written by the user, the associated Paxos state and a learned bit where true means this log entry's value has been agreed. Therefore, a replica in our implementation is both an acceptor and a learner.

Reaching consensus for a single log entry

A Paxos round can help all replicas reach consensus on a single log entry's value. It has two phases: a promise phase and a write phase. Note that we are using slightly different terminology from the original Paxos paper. In our implementation, the prepare and accept phases in the original paper are referred to as the promise and write phases, respectively. Consequently, a prepare request (response) is referred to as a promise request (response), and an accept request (response) is referred to as a write request (response).

To append value X to the log at position p, the coordinator first broadcasts a promise request to all replicas with proposal number n, asking replicas to promise that they will not respond to any request (promise/write request) with a proposal number lower than n. We assume that n is higher than any other previously used proposal number, and will explain how we do this later.

When receiving the promise request, each replica checks its Paxos state to decide if it can safely respond to the request, depending on the promises it has previously given out. If the replica is able to give the promise (i.e., passes the proposal number check), it will first persist its promise (the proposal number n) on disk and reply with a promise response. If the replica has been previously written (i.e., accepted a write request), it needs to include the previously written value along with the proposal number used in that write request into the promise response it's about to send out.

Upon receiving promise responses from a quorum of replicas, the coordinator first checks if there exist any previously written value from those responses. The append operation cannot continue if a previously written value is found because it's likely that a value has already been agreed on for that log entry. This is one of the key ideas in Paxos: restrict the value that can be written to ensure consistency.

If no previous written value is found, the coordinator broadcasts a write request to all replicas with value X and proposal number n. On receiving the write request, each replica checks the promise it has given again, and replies with a write response if the write request's proposal number is equal to or larger than the proposal number it has promised. Once the coordinator receives write responses from a quorum of replicas, the append operation succeeds.

Optimizing append latency using Multi-Paxos

One naive solution to implement a replicated log is to run a full Paxos round (promise phase and write phase) for each log entry. As discussed in the original Paxos paper, if the leader is relatively stable, Multi-Paxos can be used to eliminate the need for the promise phase for most of the append operations, resulting in improved performance.

To do that, we introduce a new type of promise request called an implicit promise request. An implicit promise request can be viewed as a batched promise request for a (potentially infinite) set of log entries. Broadcasting an implicit promise request is conceptually equivalent to broadcasting a promise request for every log entry whose value has not yet been agreed. If the implicit promise request broadcasted by a coordinator gets accepted by a quorum of replicas, this coordinator is no longer required to run the promise phase if it wants to append to a log entry whose value has not yet been agreed because the promise phase has already been done in batch. The coordinator in this case is therefore called elected (a.k.a., the leader), and has exclusive access to the replicated log. An elected coordinator may be demoted (or lose exclusive access) if another coordinator broadcasts an implicit promise request with a higher proposal number.

One question remaining is how can we find out those log entries whose values have not yet been agreed. We have a very simple solution: if a replica accepts an implicit promise request, it will include its largest known log position in the response. An elected coordinator will only append log entries at positions larger than p, where p is greater than any log position seen in these responses.

Multi-Paxos has better performance if the leader is stable. The replicated log itself does not perform leader election. Instead, we rely on the user of the replicated log to choose a stable leader. For example, Aurora uses ZooKeeper to elect the leader.

Enabling local reads

As discussed above, in our implementation, each replica is both an acceptor and a learner. Treating each replica as a learner allows us to do local reads without involving other replicas. When a log entry's value has been agreed, the coordinator will broadcast a learned message to all replicas. Once a replica receives the learned message, it will set the learned bit in the corresponding log entry, indicating the value of that log entry has been agreed. We say a log entry is "learned" if its learned bit is set. The coordinator does not have to wait for replicas' acknowledgments.

To perform a read, the log reader will directly look up the underlying local replica. If the corresponding log entry is learned, the reader can just return the value to the user. Otherwise, a full Paxos round is needed to discover the agreed value. We always make sure that the replica co-located with the elected coordinator always has all log entries learned. We achieve that by running full Paxos rounds for those unlearned log entries after the coordinator is elected.

Reducing log size using garbage collection

In case the log grows large, the application has the choice to truncate the log. To perform a truncation, we append a special log entry whose value is the log position to which the user wants to truncate the log. A replica can actually truncate the log once this special log entry has been learned.

Unique proposal number

Many of the Paxos research papers assume that each proposal number is globally unique, and a coordinator can always come up with a proposal number that is larger than any other proposal numbers in the system. However, implementing this is not trivial, especially in a distributed environment. Some researchers suggest concatenating a globally unique server id to each proposal number. But it is still not clear how to generate a globally unique id for each server.

Our solution does not make the above assumptions. A coordinator can use an arbitrary proposal number initially. During the promise phase, if a replica knows a proposal number higher than the proposal number used by the coordinator, it will send the largest known proposal number back to the coordinator. The coordinator will retry the promise phase with a higher proposal number.

To avoid livelock (e.g., when two coordinators completing), we inject a randomly delay between T and 2T before each retry. T has to be chosen carefully. On one hand, we want T >> broadcast time such that one coordinator usually times out and wins before others wake up. On the other hand, we want T to be as small as possible such that we can reduce the wait time. Currently, we use T = 100ms. This idea is actually borrowed from Raft.

Automatic replica recovery

The algorithm described above has a critical vulnerability: if a replica loses its durable state (i.e., log files) due to either disk failure or operational error, that replica may cause inconsistency in the log if it is simply restarted and re-added to the group. The operator needs to stop the application on all hosts, copy the log files from the leader's host, and then restart the application. Note that the operator cannot copy the log files from an arbitrary replica because copying an unlearned log entry may falsely assemble a quorum for an incorrect value, leading to inconsistency.

To avoid the need for operator intervention in this situation, the Mesos replicated log includes support for auto recovery. As long as a quorum of replicas is working properly, the users of the application won't notice any difference.

Non-voting replicas

To enable auto recovery, a key insight is that a replica that loses its durable state should not be allowed to respond to requests from coordinators after restart. Otherwise, it may introduce inconsistency in the log as it could have accepted a promise/write request which it would not have accepted if its previous Paxos state had not been lost.

To solve that, we introduce a new status variable for each replica. A normal replica is said in VOTING status, meaning that it is allowed to respond to requests from coordinators. A replica with no persisted state is put in EMPTY status by default. A replica in EMPTY status is not allowed to respond to any request from coordinators.

A replica in EMPTY status will be promoted to VOTING status if the following two conditions are met:

1. a sufficient amount of missing log entries are recovered such that if other replicas fail, the remaining replicas can recover all the learned log entries, and
2. its future responses to a coordinator will not break any of the promises (potentially lost) it has given out.

In the following, we discuss how we achieve these two conditions.

Catch-up

To satisfy the above two conditions, a replica needs to perform catch-up to recover lost states. In other words, it will run Paxos rounds to find out those log entries whose values that have already been agreed. The question is how many log entries the local replica should catch-up before the above two conditions can be satisfied.

We found that it is sufficient to catch-up those log entries from position begin to position end where begin is the smallest position seen in a quorum of VOTING replicas and end is the largest position seen in a quorum of VOTING replicas.

Here is our correctness argument. For a log entry at position e where e is larger than end, obviously no value has been agreed on. Otherwise, we should find at least one VOTING replica in a quorum of replicas such that its end position is larger than end. For the same reason, a coordinator should not have collected enough promises for the log entry at position e. Therefore, it's safe for the recovering replica to respond requests for that log entry. For a log entry at position b where b is smaller than begin, it should have already been truncated and the truncation should have already been agreed. Therefore, allowing the recovering replica to respond requests for that position is also safe.

Auto initialization

Since we don't allow an empty replica (a replica in EMPTY status) to respond to requests from coordinators, that raises a question for bootstrapping because initially, each replica is empty. The replicated log provides two choices here. One choice is to use a tool (mesos-log) to explicitly initialize the log on each replica by setting the replica's status to VOTING, but that requires an extra step when setting up an application.

The other choice is to do automatic initialization. Our idea is: we allow a replica in EMPTY status to become VOTING immediately if it finds all replicas are in EMPTY status. This is based on the assumption that the only time all replicas are in EMPTY status is during start-up. This may not be true if a catastrophic failure causes all replicas to lose their durable state, and that's exactly the reason we allow conservative users to disable auto-initialization.

To do auto-initialization, if we use a single-phase protocol and allow a replica to directly transit from EMPTY status to VOTING status, we may run into a state where we cannot make progress even if all replicas are in EMPTY status initially. For example, say the quorum size is 2. All replicas are in EMPTY status initially. One replica will first set its status to VOTING because if finds all replicas are in EMPTY status. After that, neither the VOTING replica nor the EMPTY replicas can make progress. To solve this problem, we use a two-phase protocol and introduce an intermediate transient status (STARTING) between EMPTY and VOTING status. A replica in EMPTY status can transit to STARTING status if it finds all replicas are in either EMPTY or STARTING status. A replica in STARTING status can transit to VOTING status if it finds all replicas are in either STARTING or VOTING status. In that way, in our previous example, all replicas will be in STARTING status before any of them can transit to VOTING status.

Non-leading VOTING replica catch-up

Starting with Mesos 1.5.0 it is possible to perform eventually consistent reads from a non-leading VOTING log replica. This makes possible to do additional work on non-leading framework replicas, e.g. offload some reading from a leader to standbys reduce failover time by keeping in-memory storage represented by the replicated log "hot".

To serve eventually consistent reads a replica needs to perform catch-up to recover the latest log state in a manner similar to how it is done during EMPTY replica recovery. After that the recovered positions can be replayed without fear of seeing "holes".

A truncation can take place during the non-leading replica catch-up. The replica may try to fill the truncated position if truncation happens after the replica has recovered begin and end positions, which may lead to producing inconsistent data during log replay. In order to protect against it we use a special tombstone flag that signals to the replica that the position was truncated and begin needs to be adjusted. The replica is not blocked from truncations during or after catching-up, which means that the user may need to retry the catch-up procedure if positions that were recovered became truncated during log replay.

Future work

Currently, replicated log does not support dynamic quorum size change, also known as reconfiguration. Supporting reconfiguration would allow us more easily to add, move or swap hosts for replicas. We plan to support reconfiguration in the future.

Agent Recovery

If the mesos-agent process on a host exits (perhaps due to a Mesos bug or because the operator kills the process while upgrading Mesos), any executors/tasks that were being managed by the mesos-agent process will continue to run.

By default, all the executors/tasks that were being managed by the old mesos-agent process are expected to gracefully exit on their own, and will be shut down after the agent restarted if they did not.

However, if a framework enabled checkpointing when it registered with the master, any executors belonging to that framework can reconnect to the new mesos-agent process and continue running uninterrupted. Hence, enabling framework checkpointing allows tasks to tolerate Mesos agent upgrades and unexpected mesos-agent crashes without experiencing any downtime.

Agent recovery works by having the agent checkpoint information about its own state and about the tasks and executors it is managing to local disk, for example the SlaveInfo, FrameworkInfo and ExecutorInfo messages or the unacknowledged status updates of running tasks.

When the agent restarts, it will verify that its current configuration, set from the environment variables and command-line flags, is compatible with the checkpointed information and will refuse to restart if not.

A special case occurs when the agent detects that its host system was rebooted since the last run of the agent: The agent will try to recover its previous ID as usual, but if that fails it will actually erase the information of the previous run and will register with the master as a new agent.

Note that executors and tasks that exited between agent shutdown and restart are not automatically restarted during agent recovery.

Framework Configuration

A framework can control whether its executors will be recovered by setting the checkpoint flag in its FrameworkInfo when registering with the master. Enabling this feature results in increased I/O overhead at each agent that runs tasks launched by the framework. By default, frameworks do not checkpoint their state.

Agent Configuration

Four configuration flags control the recovery behavior of a Mesos agent:

• strict: Whether to do agent recovery in strict mode [Default: true].

  • If strict=true, all recovery errors are considered fatal.
  • If strict=false, any errors (e.g., corruption in checkpointed data) during recovery are ignored and as much state as possible is recovered.

• reconfiguration_policy: Which kind of configuration changes are accepted when trying to recover [Default: equal].

  • If reconfiguration_policy=equal, no configuration changes are accepted.
  • If reconfiguration_policy=additive, the agent will allow the new configuration to contain additional attributes, increased resourced or an additional fault domain. For a more detailed description, see this.

• recover: Whether to recover status updates and reconnect with old executors [Default: reconnect]

  • If recover=reconnect, reconnect with any old live executors, provided the executor's framework enabled checkpointing.
  • If recover=cleanup, kill any old live executors and exit. Use this option when doing an incompatible agent or executor upgrade! NOTE: If no checkpointing information exists, no recovery is performed and the agent registers with the master as a new agent.

• recovery_timeout: Amount of time allotted for the agent to recover [Default: 15 mins].

  • If the agent takes longer than recovery_timeout to recover, any executors that are waiting to reconnect to the agent will self-terminate. NOTE: If none of the frameworks have enabled checkpointing, the executors and tasks running at an agent die when the agent dies and are not recovered.

A restarted agent should reregister with master within a timeout (75 seconds by default: see the --max_agent_ping_timeouts and --agent_ping_timeout configuration flags). If the agent takes longer than this timeout to reregister, the master shuts down the agent, which in turn will shutdown any live executors/tasks.

Therefore, it is highly recommended to automate the process of restarting an agent, e.g. using a process supervisor such as monit or systemd.

Known issues with systemd and process lifetime

There is a known issue when using systemd to launch the mesos-agent. A description of the problem can be found in MESOS-3425 and all relevant work can be tracked in the epic MESOS-3007.

This problem was fixed in Mesos 0.25.0 for the mesos containerizer when cgroups isolation is enabled. Further fixes for the posix isolators and docker containerizer are available in 0.25.1, 0.26.1, 0.27.1, and 0.28.0.

It is recommended that you use the default KillMode for systemd processes, which is control-group, which kills all child processes when the agent stops. This ensures that "side-car" processes such as the fetcher and perf are terminated alongside the agent. The systemd patches for Mesos explicitly move executors and their children into a separate systemd slice, dissociating their lifetime from the agent. This ensures the executors survive agent restarts.

The following excerpt of a systemd unit configuration file shows how to set the flag explicitly:

[Service]
ExecStart=/usr/bin/mesos-agent
KillMode=control-cgroup

Framework Rate Limiting

Framework rate limiting is a feature introduced in Mesos 0.20.0.

What is Framework Rate Limiting

In a multi-framework environment, this feature aims to protect the throughput of high-SLA (e.g., production, service) frameworks by having the master throttle messages from other (e.g., development, batch) frameworks.

To throttle messages from a framework, the Mesos cluster operator sets a qps (queries per seconds) value for each framework identified by its principal (You can also throttle a group of frameworks together but we'll assume individual frameworks in this doc unless otherwise stated; see the RateLimits Protobuf definition and the configuration notes below). The master then promises not to process messages from that framework at a rate above qps. The outstanding messages are stored in memory on the master.

Rate Limits Configuration

The following is a sample config file (in JSON format) which could be specified with the --rate_limits master flag.

{
  "limits": [
    {
      "principal": "foo",
      "qps": 55.5
      "capacity": 100000
    },
    {
      "principal": "bar",
      "qps": 300
    },
    {
      "principal": "baz",
    }
  ],
  "aggregate_default_qps": 333,
  "aggregate_default_capacity": 1000000
}

In this example, framework foo is throttled at the configured qps and capacity, framework bar is given unlimited capacity and framework baz is not throttled at all. If there is a fourth framework qux or a framework without a principal connected to the master, it is throttled by the rules aggregate_default_qps and aggregate_default_capacity.

Configuration Notes

Below are the fields in the JSON configuration.

• principal: (Required) uniquely identifies the entity being throttled or given unlimited rate explicitly.
  • It should match the framework's FrameworkInfo.principal (See definition).
  • You can have multiple frameworks use the same principal (e.g., some Mesos frameworks launch a new framework instance for each job), in which case the combined traffic from all frameworks using the same principal are throttled at the specified QPS.
• qps: (Optional) queries per second, i.e., the rate.
  • Once set, the master guarantees that it does not process messages from this principal higher than this rate. However the master could be slower than this rate, especially if the specified rate is too high.
  • To explicitly give a framework unlimited rate (i.e., not throttling it), add an entry to limits without the qps.
• capacity: (Optional) The number of outstanding messages frameworks of this principal can put on the master. If not specified, this principal is given unlimited capacity. Note that it is possible the queued messages use too much memory and cause the master to OOM if the capacity is set too high or not set.
  • NOTE: If qps is not specified, capacity is ignored.
• Use aggregate_default_qps and aggregate_default_capacity to safeguard the master from unspecified frameworks. All the frameworks not specified in limits get this default rate and capacity.
  • The rate and capacity are aggregate values for all of them, i.e., their combined traffic is throttled together.
  • Same as above, if aggregate_default_qps is not specified, aggregate_default_capacity is ignored.
  • If these fields are not present, the unspecified frameworks are not throttled. This is an implicit way of giving frameworks unlimited rate compared to the explicit way above (using an entry in limits with only the principal). We recommend using the explicit option especially when the master does not require authentication to prevent unexpected frameworks from overwhelming the master.

Using Framework Rate Limiting

Monitoring Framework Traffic

While a framework is registered with the master, the master exposes counters for all messages received and processed from that framework at its metrics endpoint: http://<master>/metrics/snapshot. For instance, framework foo has two message counters frameworks/foo/messages_received and frameworks/foo/messages_processed. Without framework rate limiting the two numbers should differ by little or none (because messages are processed ASAP) but when a framework is being throttled the difference indicates the outstanding messages as a result of the throttling.

By continuously monitoring the counters, you can derive the rate messages arrive and how fast the message queue length for the framework is growing (if it is throttled). This should depict the characteristics of the framework in terms of network traffic.

Configuring Rate Limits

Since the goal for framework rate limiting is to prevent low-SLA frameworks from using too much resources and not to model their traffic and behavior as precisely as possible, you can start by using large qps values to throttle them. The fact that they are throttled (regardless of the configured qps) is already effective in giving messages from high-SLA frameworks higher priority because they are processed ASAP.

To calculate how much capacity the master can handle, you need to know the memory limit for the master process, the amount of memory it typically uses to serve similar workload without rate limiting (e.g., use ps -o rss $MASTER_PID) and average sizes of the framework messages (queued messages are stored as serialized Protocol Buffers with a few additional fields) and you should sum up all capacity values in the config. However since this kind of calculation is imprecise, you should start with small values that tolerate reasonable temporary framework burstiness but far from the memory limit to leave enough headroom for the master and frameworks that don't have limited capacity.

Handling "Capacity Exceeded" Error

When a framework exceeds the capacity, a FrameworkErrorMessage is sent back to the framework which will abort the scheduler driver and invoke the error() callback. It doesn't kill any tasks or the scheduler itself. The framework developer can choose to restart or failover the scheduler instance to remedy the consequences of dropped messages (unless your framework doesn't assume all messages sent to the master are processed).

After version 0.20.0 we are going to iterate on this feature by having the master send an early alert when the message queue for this framework starts to build up (MESOS-1664, consider it a "soft limit"). The scheduler can react by throttling itself (to avoid the error message) or ignoring this alert if it's a temporary burst by design.

Before the early alerting is implemented we don't recommend using the rate limiting feature to throttle production frameworks for now unless you are sure about the consequences of the error message. Of course it's OK to use it to protect production frameworks by throttling other frameworks and it doesn't have any effect on the master if it's not explicitly enabled.

Performing Node Maintenance in a Mesos Cluster

Operators regularly need to perform maintenance tasks on machines that comprise a Mesos cluster. Most Mesos upgrades can be done without affecting running tasks, but there are situations where maintenance may affect running tasks. For example:

• Hardware repair
• Kernel upgrades
• Agent upgrades (e.g., adjusting agent attributes or resources)

Before performing maintenance on an agent node in a Mesos cluster, it is typically desirable to gracefully migrate tasks away from the node beforehand in order to minimize service disruption when the machine is taken down. Mesos provides several ways to accomplish this migration:

• Automatic agent draining, which does not explicitly require cooperation from schedulers
• Manual node draining, which allows operators to exercise precise control over the task draining process
• Maintenance primitives, which permit complex coordination but do require that schedulers react to the maintenance-related messages that they receive

Automatic Node Draining

Node draining was added to provide a simple method for operators to drain tasks from nodes on which they plan to perform maintenance, without requiring that schedulers implement support for any maintenance-specific messages.

Initiating draining will cause all tasks on the target agent node to receive a kill event immediately, assuming the agent is currently reachable. If the agent is unreachable, initiation of the kill event will be delayed until the agent is reachable by the master again. When the tasks receive a kill event, a SIGTERM signal will be sent to the task to begin the killing process. Depending on the particular task's behavior, this signal may be sufficient to terminate it. Some tasks may use this signal to begin the process of graceful termination, which may take some time. After some delay, a SIGKILL signal will be sent to the task, which forcefully terminates the task if it is still running. The delay between the SIGTERM and SIGKILL signals is determined by the length of the task's kill grace period. If no grace period is set for the task, a default value of several seconds will be used.

Initiating Draining on a Node

To begin draining an agent, issue the operator API DRAIN_AGENT call to the master:

$ curl -X POST -d '{"type": "DRAIN_AGENT", "drain_agent": {"agent_id": {"value": "<mesos-agent-id>"}}}' masterhost:5050/api/v1

This will immediately begin the process of killing all tasks on the agent. Once draining has begun, it cannot be cancelled. To monitor the progress of the draining process, you can inspect the state of the agent via the master operator API GET_STATE or GET_AGENTS calls:

$ curl -X POST -d '{"type": "GET_AGENTS"}' masterhost:5050/api/v1

Locate the relevant agent and inspect its drain_info.state field. While draining, the state will be DRAINING. When all tasks on the agent have terminated, all their terminal status updates have been acknowledged by the schedulers, and all offer operations on the agent have finished, draining is complete and the agent's drain state will transition to DRAINED. At this point, the node may be taken down for maintenance.

Options for Automatic Node Draining

You may set an upper bound on the kill grace period of draining tasks by specifying the max_grace_period option when draining:

$ curl -X POST -d '{"type": "DRAIN_AGENT", "drain_agent": {"agent_id": {"value": "<mesos-agent-id>"}, "max_grace_period": "10mins"}}' masterhost:5050/api/v1

In cases where you know that the node being drained will not return after draining is complete, and you would like it to be automatically permanently removed from the cluster, you may specify the mark_gone option:

$ curl -X POST -d '{"type": "DRAIN_AGENT", "drain_agent": {"agent_id": {"value": "<mesos-agent-id>"}, "mark_gone": true}}' masterhost:5050/api/v1

This can be useful, for example, in the case of autoscaled cloud instances, where an instance is being scaled down and will never return. This is equivalent to issuing the MARK_AGENT_GONE call on the agent immediately after it finishes draining. WARNING: draining with the mark_gone option is irreversible, and results in the loss of all local persistent data on the agent node. Use this option with caution!

Reactivating a Node After Maintenance

Once maintenance on an agent is complete, it must be reactivated so that it can reregister with the master and rejoin the cluster. You may use the master operator API REACTIVATE_AGENT call to accomplish this:

$ curl -X POST -d '{"type": "REACTIVATE_AGENT", "reactivate_agent": {"agent_id": {"value": "<mesos-agent-id>"}}}' masterhost:5050/api/v1

Manual Node Draining

If you require greater control over the draining process, you may be able to drain the agent manually using both the Mesos operator API as well as APIs exposed by the schedulers running tasks on the agent.

Deactivating an Agent

The first step in the manual draining process is agent deactivation, which prevents new tasks from launching on the target agent:

$ curl -X POST -d '{"type": "DEACTIVATE_AGENT", "deactivate_agent": {"agent_id": {"value": "<mesos-agent-id>"}}}' masterhost:5050/api/v1

If you receive a 200 OK response, then the agent has been deactivated. You can confirm the deactivation state of any agent by inspecting its deactivated field in the response of the master operator API GET_STATE or GET_AGENTS calls. Once the agent is deactivated, you can use the APIs exposed by the schedulers responsible for the tasks running on the agent to kill those tasks manually. To verify that all tasks on the agent have terminated and their terminal status updates have been acknowledged by the schedulers, ensure that the pending_tasks, queued_tasks, and launched_tasks fields in the response to the GET_TASKS agent operator API call are empty:

$ curl -X POST -d '{"type": "GET_TASKS"}' agenthost:5051/api/v1

If you are making use of volumes backed by network storage on the target agent, it's possible that there may be a long-running offer operation on the agent which has not yet finished. To check if this is the case, issue the agent operator API GET_OPERATIONS call to the agent:

$ curl -X POST -d '{"type": "GET_OPERATIONS"}' agenthost:5051/api/v1

If any operations have a latest_status with a state of OPERATION_PENDING, you should wait for them to finish before taking down the node. Unfortunately, it is not possible to cancel or forcefully terminate such storage operations. If such an operation becomes stuck in the pending state, you should inspect the relevant storage backend for any issues.

Once all tasks on the agent have terminated and all offer operations are finished, the node may be taken down for maintenance. Once maintenance is complete, the procedure for reactivating the node is the same as that detailed in the section on automatic node draining.

Maintenance Primitives

Frameworks require visibility into any actions that disrupt cluster operation in order to meet Service Level Agreements or to ensure uninterrupted services for their end users. Therefore, to reconcile the requirements of frameworks and operators, frameworks must be aware of planned maintenance events and operators must be aware of frameworks' ability to adapt to maintenance. Maintenance primitives add a layer to facilitate communication between the frameworks and operator.

Terminology

For the purpose of this section, an "Operator" is a person, tool, or script that manages a Mesos cluster.

Maintenance primitives add several new concepts to Mesos. Those concepts are:

• Maintenance: An operation that makes resources on a machine unavailable, either temporarily or permanently.
• Maintenance window: A set of machines and an associated time interval during which some maintenance is planned on those machines.
• Maintenance schedule: A list of maintenance windows. A single machine may only appear in a schedule once.
• Unavailability: An operator-specified interval, defined by a start time and duration, during which an associated machine may become unavailable. In general, no assumptions should be made about the availability of the machine (or resources) after the unavailability.
• Drain: An interval between the scheduling of maintenance and when the machine(s) become unavailable. Offers sent with resources from draining machines will contain unavailability information. Frameworks running on draining machines will receive inverse offers (see next). Frameworks utilizing resources on affected machines are expected either to take preemptive steps to prepare for the unavailability; or to communicate the framework's inability to conform to the maintenance schedule.
• Inverse offer: A communication mechanism for the master to ask for resources back from a framework. This notifies frameworks about any unavailability and gives frameworks a mechanism to respond about their ability to comply. Inverse offers are similar to offers in that they can be accepted, declined, re-offered, and rescinded.

Note: Unavailability and inverse offers are not specific to maintenance. The same concepts can be used for non-maintenance goals, such as reallocating resources or resource preemption.

How does it work?

Maintenance primitives were introduced in Mesos 0.25.0. Several machine maintenance modes were also introduced. Those modes are illustrated below.

All mode transitions must be initiated by the operator. Mesos will not change the mode of any machine, regardless of the estimate provided in the maintenance schedule.

Scheduling maintenance

A machine is transitioned from Up mode to Draining mode as soon as it is scheduled for maintenance. To transition a machine into Draining mode, an operator constructs a maintenance schedule as a JSON document and posts it to the /maintenance/schedule HTTP endpoint on the Mesos master. Each Mesos cluster has a single maintenance schedule; posting a new schedule replaces the previous schedule, if any.

See the definition of a maintenance::Schedule and of Unavailability.

In a production environment, the schedule should be constructed to ensure that enough agents are operational at any given point in time to ensure uninterrupted service by the frameworks.

For example, in a cluster of three machines, the operator might schedule two machines for one hour of maintenance, followed by another hour for the last machine. The timestamps for unavailability are expressed in nanoseconds since the Unix epoch (note that making reliable use of maintenance primitives requires that the system clocks of all machines in the cluster are roughly synchronized).

The schedule might look like:

{
  "windows" : [
    {
      "machine_ids" : [
        { "hostname" : "machine1", "ip" : "10.0.0.1" },
        { "hostname" : "machine2", "ip" : "10.0.0.2" }
      ],
      "unavailability" : {
        "start" : { "nanoseconds" : 1443830400000000000 },
        "duration" : { "nanoseconds" : 3600000000000 }
      }
    }, {
      "machine_ids" : [
        { "hostname" : "machine3", "ip" : "10.0.0.3" }
      ],
      "unavailability" : {
        "start" : { "nanoseconds" : 1443834000000000000 },
        "duration" : { "nanoseconds" : 3600000000000 }
      }
    }
  ]
}

The operator can then post the schedule to the master's /maintenance/schedule endpoint:

curl http://localhost:5050/maintenance/schedule \
  -H "Content-type: application/json" \
  -X POST \
  -d @schedule.json

The machines in a maintenance schedule do not need to be registered with the Mesos master at the time when the schedule is set. The operator may add a machine to the maintenance schedule prior to launching an agent on the machine. For example, this can be useful to prevent a faulty machine from launching an agent on boot.

Note: Each machine in the maintenance schedule should have as complete information as possible. In order for Mesos to recognize an agent as coming from a particular machine, both the hostname and ip fields must match. Any omitted data defaults to the empty string "". If there are multiple hostnames or IPs for a machine, the machine's fields need to match what the agent announces to the master. If there is any ambiguity in a machine's configuration, the operator should use the --hostname and --ip options when starting agents.

The master checks that a maintenance schedule has the following properties:

• Each maintenance window in the schedule must have at least one machine and a specified unavailability interval.
• Each machine must only appear in the schedule once.
• Each machine must have at least a hostname or IP included. The hostname is not case-sensitive.
• All machines that are in Down mode must be present in the schedule. This is required because this endpoint does not handle the transition from Down mode to Up mode.

If any of these properties are not met, the maintenance schedule is rejected with a corresponding error message and the master's state is not changed.

To update the maintenance schedule, the operator should first read the current schedule, make any necessary changes, and then post the modified schedule. The current maintenance schedule can be obtained by sending a GET request to the master's /maintenance/schedule endpoint.

To cancel the maintenance schedule, the operator should post an empty schedule.

Draining mode

As soon as a schedule is posted to the Mesos master, the following things occur:

• The schedule is stored in the replicated log. This means the schedule is persisted in case of master failover.
• All machines in the schedule are immediately transitioned into Draining mode. The mode of each machine is also persisted in the replicated log.
• All frameworks using resources on affected agents are immediately notified. Existing offers from the affected agents are rescinded and re-sent with additional unavailability data. All frameworks using resources from the affected agents are given inverse offers.
• New offers from the affected agents will also include the additional unavailability data.

Frameworks should use this additional information to schedule tasks in a maintenance-aware fashion. Exactly how to do this depends on the design requirements of each scheduler, but tasks should typically be scheduled in a way that maximizes utilization but that also attempts to vacate machines before that machine's advertised unavailability period occurs. A scheduler might choose to place long-running tasks on machines with no unavailability, or failing that, on machines whose unavailability is the furthest away.

How a framework responds to an inverse offer indicates its ability to conform to the maintenance schedule. Accepting an inverse offer communicates that the framework is okay with the current maintenance schedule, given the current state of the framework's resources. The master and operator should interpret acceptance as a best-effort promise by the framework to free all the resources contained in the inverse offer before the start of the unavailability interval. Declining an inverse offer is an advisory notice to the operator that the framework is unable or unlikely to meet to the maintenance schedule.

For example:

• A data store may choose to start a new replica if one of its agents is scheduled for maintenance. The data store should accept an inverse offer if it can reasonably copy the data on the machine to a new host before the unavailability interval described in the inverse offer begins. Otherwise, the data store should decline the offer.
• A stateful task on an agent with an impending unavailability may be migrated to another available agent. If the framework has sufficient resources to do so, it would accept any inverse offers. Otherwise, it would decline them.

A framework can use a filter to control when it wants to be contacted again with an inverse offer. This is useful since future circumstances may change the viability of the maintenance schedule. The filter for inverse offers is identical to the existing mechanism for re-offering offers to frameworks.

Note: Accepting or declining an inverse offer does not result in immediate changes in the maintenance schedule or in the way Mesos acts. Inverse offers only represent extra information that frameworks may find useful. In the same manner, rejecting or accepting an inverse offer is a hint for an operator. The operator may or may not choose to take that hint into account.

Starting maintenance

The operator starts maintenance by posting a list of machines to the /machine/down HTTP endpoint. The list of machines is specified in JSON format; each element of the list is a MachineID.

For example, to start maintenance on two machines:

[
  { "hostname" : "machine1", "ip" : "10.0.0.1" },
  { "hostname" : "machine2", "ip" : "10.0.0.2" }
]

curl http://localhost:5050/machine/down \
  -H "Content-type: application/json" \
  -X POST \
  -d @machines.json

The master checks that a list of machines has the following properties:

• The list of machines must not be empty.
• Each machine must only appear once.
• Each machine must have at least a hostname or IP included. The hostname is not case-sensitive.
• If a machine's IP is included, it must be correctly formed.
• All listed machines must be present in the schedule.

If any of these properties are not met, the operation is rejected with a corresponding error message and the master's state is not changed.

The operator can start maintenance on any machine that is scheduled for maintenance. Machines that are not scheduled for maintenance cannot be directly transitioned from Up mode into Down mode. However, the operator may schedule a machine for maintenance with a timestamp equal to the current time or in the past, and then immediately start maintenance on that machine.

This endpoint can be used to start maintenance on machines that are not currently registered with the Mesos master. This can be useful if a machine has failed and the operator intends to remove it from the cluster; starting maintenance on the machine prevents the machine from being accidentally rebooted and rejoining the Mesos cluster.

The operator must explicitly transition a machine from Draining to Down mode. That is, Mesos will keep a machine in Draining mode even if the unavailability window arrives or passes. This means that the operation of the machine is not disrupted in any way and offers (with unavailability information) are still sent for this machine.

When maintenance is triggered by the operator, all agents on the machine are told to shutdown. These agents are removed from the master, which means that a TASK_LOST status update will be sent for every task running on each of those agents. The scheduler driver's slaveLost callback will also be invoked for each of the removed agents. Any agents on machines in maintenance are also prevented from reregistering with the master in the future (until maintenance is completed and the machine is brought back up).

Completing maintenance

When maintenance is complete or if maintenance needs to be cancelled, the operator can stop maintenance. The process is very similar to starting maintenance (same validation criteria as the previous section). The operator posts a list of machines to the master's /machine/up endpoint:

[
  { "hostname" : "machine1", "ip" : "10.0.0.1" },
  { "hostname" : "machine2", "ip" : "10.0.0.2" }
]

curl http://localhost:5050/machine/up \
  -H "Content-type: application/json" \
  -X POST \
  -d @machines.json

Note: The duration of the maintenance window, as indicated by the "unavailability" field in the maintenance schedule, is a best-effort guess made by the operator. Stopping maintenance before the end of the unavailability interval is allowed, as is stopping maintenance after the end of the unavailability interval. Machines are never automatically transitioned out of maintenance.

Frameworks are informed about the completion or cancellation of maintenance when offers from that machine start being sent. There is no explicit mechanism for notifying frameworks when maintenance has finished. After maintenance has finished, new offers are no longer tagged with unavailability and inverse offers are no longer sent. Also, agents running on the machine will be allowed to register with the Mesos master.

Viewing maintenance status

The current maintenance status (Up, Draining, or Down) of each machine in the cluster can be viewed by accessing the master's /maintenance/status HTTP endpoint. For each machine that is Draining, this endpoint also includes the frameworks' responses to inverse offers for resources on that machine. For more information, see the format of the ClusterStatus message.

NOTE: The format of the data returned by this endpoint may change in a future release of Mesos.

title: Apache Mesos - Upgrading Mesos layout: documentation

Upgrading Mesos

This document serves as a guide for users who wish to upgrade an existing Mesos cluster. Some versions require particular upgrade techniques when upgrading a running cluster. Some upgrades will have incompatible changes.

Overview

This section provides an overview of the changes for each version (in particular when upgrading from the next lower version). For more details please check the respective sections below.

We categorize the changes as follows:

A New feature/behavior
C Changed feature/behavior
D Deprecated feature/behavior
R Removed feature/behavior

Upgrading from 1.9.x to 1.10.x

• The canonical name for the environment variable LIBPROCESS_SSL_VERIFY_CERT was changed to LIBPROCESS_SSL_VERIFY_SERVER_CERT. The canonical name for the environment variable LIBPROCESS_SSL_REQUIRE_CERT was changed to LIBPROCESS_SSL_REQUIRE_CLIENT_CERT. The old names will continue to work as before, but operators are encouraged to update their configuration to reduce confusion.

• The Mesos agent's cgroups_enable_cfs flag previously controlled whether or not CFS quota would be used for all tasks on the agent. Resource limits have been added to tasks, and when a CPU limit is specified on a task, the agent will now apply a CFS quota regardless of the value of cgroups_enable_cfs.

• The Mesos agent now requires the new TASK_RESOURCE_LIMITS feature. This capability is set by default, but if the --agent_features flag is specified explicitly, TASK_RESOURCE_LIMITS must be included.

• Authorizers now must implement a method getApprover(...) (see the authorization documentation and MESOS-10056) that returns ObjectApprovers that are valid throughout their whole lifetime. Keeping the state of an ObjectApprover up-to-date becomes a responsibility of the authorizer. This is a breaking change for authorizer modules.

• The field pending_tasks in GetTasks master API call has been deprecated. From now on, this field will be empty. Moreover, the notion of tasks pending authorization no longer exists (see MESOS-10056).

• Allocator interface has been changed to supply allocator with information on resources actually consumed by frameworks. A method transitionOfferedToAllocated(...) has been added and the signature of recoverResources(...) has been extended. Note that allocators must implement these new/extended method signatures, but are free to ignore resource consumption data provided by master.

Upgrading from 1.8.x to 1.9.x

• A new DRAINING state has been added to Mesos agents. Once an agent is draining, all tasks running on that agent are gracefully killed and no offers for that agent are sent to schedulers, preventing the launching of new tasks. Operators can put an agent into DRAINING state by using the DRAIN_AGENT operator API call. See docs/maintenance for details.

• The Mesos agent now requires the new AGENT_DRAINING feature. This capability is set by default, but if the --agent_features flag is specified explicitly, AGENT_DRAINING must be included.

• A new linux/nnp isolator has been added. The isolator supports setting of the no_new_privs bit in the container, preventing tasks from acquiring additional privileges.

• A new --docker_ignore_runtime flag has been added. This causes the agent to ignore any runtime configuration present in Docker images.

• A new libprocess TLS flag --hostname_validation_scheme along with the corresponding environment variable LIBPROCESS_SSL_HOSTNAME_VALIDATION_SCHEME has been added. Using this flag, users can configure the way libprocess performs hostname validation for TLS connections. See docs/ssl for details.

• The semantics of the libprocess environment variables LIBPROCESS_SSL_VERIFY_CERT and LIBPROCESS_SSL_REQUIRE_CERT have been slightly updated such that the former now only applies to client-mode and the latter only to server-mode connections. As part of this re-adjustment, the following two changes have been introduced that might require changes for operators running Mesos in unusual TLS configurations.
  • Anonymous ciphers can not be used anymore when LIBPROCESS_SSL_VERIFY_CERT is set to true. This is because the use of anonymous ciphers enables a malicious attacker to bypass certificate verification by choosing a certificate-less cipher. Users that rely on anonymous ciphers being available should make sure that LIBPROCESS_SSL_VERIFY_CERT is set to false.
  • For incoming connections, certificates are not verified unless LIBPROCESS_SSL_REQUIRE_CERT is set to true. This is because verifying the certificate can lead to false negatives, where a connection is aborted even though presenting no certificate at all would have been successfull. Users that rely on incoming connection requests presenting valid TLS certificates should make sure that the LIBPROCESS_SSL_REQUIRE_CERT option is set to true.

• The Mesos containerizer now supports configurable IPC namespace and /dev/shm. Container can be configured to have a private IPC namespace and /dev/shm or share them from its parent via the field LinuxInfo.ipc_mode, and the size of its private /dev/shm is also configurable via the field LinuxInfo.shm_size. Operators can control whether it is allowed to share host's IPC namespace and /dev/shm with top level containers via the agent flag --disallow_sharing_agent_ipc_namespace, and specify the default size of the /dev/shm for the container which has a private /dev/shm via the agent flag --default_container_shm_size.

• The SET_QUOTA and REMOVE QUOTA master calls are deprecated in favor of a new UPDATE_QUOTA master call.

• Prior to Mesos 1.9, the quota related APIs only exposed quota "guarantees" which ensured a minimum amount of resources would be available to a role. Setting guarantees also set implicit quota limits. In Mesos 1.9+, quota limits are now exposed directly.
  • Quota guarantees are now deprecated in favor of using only quota limits. Enforcement of quota guarantees required that Mesos holds back enough resources to meet all of the unsatisfied quota guarantees. Since Mesos is moving towards an optimistic offer model (to improve multi-role / multi- scheduler scalability, see MESOS-1607), it will become no longer possible to enforce quota guarantees by holding back resources. In such a model, quota limits are simple to enforce, but quota guarantees would require a complex "effective limit" propagation model to leave space for unsatisfied guarantees.
  • For these reasons, quota guarantees, while still functional in Mesos 1.9, are now deprecated. A combination of limits and priority based preemption will be simpler in an optimistic offer model.

Upgrading from 1.7.x to 1.8.x

• A new linux/seccomp isolator has been added. The isolator supports the following new agent flags:
  • --seccomp_config_dir specifies the directory path of the Seccomp profiles.
  • --seccomp_profile_name specifies the path of the default Seccomp profile relative to the seccomp_config_dir.

Upgrading from 1.6.x to 1.7.x

• A new linux/devices isolator has been added. This isolator automatically populates containers with devices that have been whitelisted with the --allowed_devices agent flag.

• A new option cgroups/all has been added to the agent flag --isolation. This allows cgroups isolator to automatically load all the local enabled cgroups subsystems. If this option is specified in the agent flag --isolation along with other cgroups related options (e.g., cgroups/cpu), those options will be just ignored.

• Added container-specific cgroups mounts under /sys/fs/cgroup to containers with image launched by Mesos containerizer.

• Previously the HOST_PATH, SANDBOX_PATH, IMAGE, SECRET, and DOCKER_VOLUME volumes were always mounted for container in read-write mode, i.e., the Volume.mode field was not honored. Now we will mount these volumes based on the Volume.mode field so framework can choose to mount the volume for the container in either read-write mode or read-only mode.

• To simplify the API for CSI-backed disk resources, the following operations and corresponding ACLs have been introduced to replace the experimental CREATE_VOLUME, CREATE_BLOCK, DESTROY_VOLUME and DESTROY_BLOCK operations:
  • CREATE_DISK to create a MOUNT or BLOCK disk resource from a RAW disk resource. The CreateMountDisk and CreateBlockDisk ACLs control which principals are allowed to create MOUNT or BLOCK disks for which roles.
  • DESTROY_DISK to reclaim a MOUNT or BLOCK disk resource back to a RAW disk resource. The DestroyMountDisk and DestroyBlockDisk ACLs control which principals are allowed to reclaim MOUNT or BLOCK disks for which roles.

• A new ViewResourceProvider ACL has been introduced to control which principals are allowed to call the GET_RESOURCE_PROVIDERS agent API.

• A new --enforce_container_ports flag has been added to toggle whether the network/ports isolator should enforce TCP ports usage limits.

• A new --gc_non_executor_container_sandboxes agent flag has been added to garbage collect the sandboxes of nested containers, which includes the tasks groups launched by the default executor. We recommend enabling the flag if you have frameworks that launch multiple task groups on the same default executor instance.

• A new --network_cni_root_dir_persist flag has been added to toggle whether the network/cni isolator should persist the network information across reboots.

• ContainerLogger module interface has been changed. The prepare() method now takes ContainerID and ContainerConfig instead.

• Isolator::recover() has been updated to take an std::vector instead of std::list of container states.

• As a result of adapting rapidjson for performance improvement, all JSON endpoints serialize differently while still conforming to the ECMA-404 spec for JSON. This means that if a client has a JSON de-serializer that conforms to ECMA-404 they will see no change. Otherwise, they may break. As an example, Mesos would previously serialize '/' as '/', but the spec does not require the escaping and rapidjson does not escape '/'.

Upgrading from 1.5.x to 1.6.x

• gRPC version 1.10+ is required to build Mesos when enabling gRPC-related features. Please upgrade your gRPC library if you are using an unbundled one.

• CSI v0.2 is now supported as experimental. Due to the incompatibility between CSI v0.1 and v0.2, the experimental support for CSI v0.1 is removed, and the operator must remove all storage local resource providers within an agent before upgrading the agent. NOTE: This is a breaking change for storage local resource providers.

• A new agent flag --fetcher_stall_timeout has been added. This flag specifies the amount of time for the container image and artifact fetchers to wait before aborting a stalled download (i.e., the speed keeps below one byte per second). NOTE: This flag only applies when downloading data from the net and does not apply to HDFS.

• The disk profile adaptor module has been changed to support CSI v0.2, and its header file has been renamed to be consistent with other modules. See disk_profile_adaptor.hpp for interface changes.

• A new agent flag --xfs_kill_containers has been added. By setting this flag, the disk/xfs isolator will now kill containers that exceed the disk limit.

Upgrading from 1.4.x to 1.5.x

• The built-in executors will now send a TASK_STARTING status update for every task they've successfully received and are about to start. The possibility of any executor sending this update has been documented since the beginning of Mesos, but prior to this version the built-in executors did not actually send it. This means that all schedulers using one of the built-in executors must be upgraded to expect TASK_STARTING updates before upgrading Mesos itself.

• A new field, limitation, was added to the TaskStatus message. This field is a TaskResourceLimitation message that describes the resources that caused a task to fail with a resource limitation reason.

• A new network/ports isolator has been added. The isolator supports the following new agent flags:
  • --container_ports_watch_interval specifies the interval at which the isolator reconciles port assignments.
  • --check_agent_port_range_only excludes ports outside the agent's range from port reconciliation.

• Agent flag --executor_secret_key has been deprecated. Operators should use --jwt_secret_key instead.

• The fields Resource.disk.source.path.root and Resource.disk.source.mount.root can now be set to relative paths to an agent's work directory. The containerizers will interpret the paths based on the --work_dir flag on an agent.

• The agent operator API call GET_CONTAINERS has been updated to support listing nested or standalone containers. One can specify the following fields in the request:
  • show_nested: Whether to show nested containers.
  • show_standalone: Whether to show standalone containers.

• A new agent flag --reconfiguration_policy has been added. By setting the value of this flag to additive, operators can allow the agent to be restarted with increased resources without requiring the agent ID to be changed. Note that if this feature is used, the master version is required to be >= 1.5 as well.

• Protobuf version 3+ is required to build Mesos. Please upgrade your Protobuf library if you are using an unbundled one.

• A new catchup() method has been added to the replicated log reader API. The method allows to catch-up positions missing in the local non-leading replica to allow safe eventually consistent reads from it. Note about backwards compatibility: In order for the feature to work correctly in presence of log truncations all log replicas need to be updated.

Upgrading from 1.3.x to 1.4.x

• If the mesos-agent kernel supports ambient capabilities (Linux 4.3 or later), the capabilities specified in the LinuxInfo.effective_capabilities message will be made ambient in the container task.

• Explicitly setting the bounding capabilities of a task independently of the effective capabilities is now supported. Frameworks can specify the task bounding capabilities by using the LinuxInfo.bounding_capabilities message. Operators can specify the default bounding capabilities using the agent --bounding_capabilities flag. This flag also specifies the maximum bounding set that a framework is allowed to specify.

• Agent is now allowed to recover its agent ID post a host reboot. This prevents the unnecessary discarding of agent ID by prior Mesos versions. Notes about backwards compatibility:
  • In case the agent's recovery runs into agent info mismatch which may happen due to resource change associated with reboot, it'll fall back to recovering as a new agent (existing behavior).
  • In other cases such as checkpointed resources (e.g. persistent volumes) being incompatible with the agent's resources the recovery will still fail (existing behavior).

• The LinuxInfo.capabilities field has been deprecated in favor of LinuxInfo.effective_capabilities.

• Changes to capability-related agent flags:
  • The agent --effective_capabilities flag has been added to specify the default effective capability set for tasks.
  • The agent --bounding_capabilities flag has been added to specify the default bounding capability set for tasks.
  • The agent --allowed-capabilities flag has been deprecated in favor of --effective_capabilities.

• The semantics of the optional resource argument passed in Allocator::updateSlave was change. While previously the passed value denoted a new amount of oversubscribed (revocable) resources on the agent, it now denotes the new amount of total resources on the agent. This requires custom allocator implementations to update their interpretation of the passed value.

• The XFS Disk Isolator now supports the --no-enforce_container_disk_quota option to efficiently measure disk resource usage without enforcing any usage limits.

• The Resources class in the internal Mesos C++ library changed its behavior to only support post-RESERVATION_REFINEMENT format. If a framework is using this internal utility, it is likely to break if the RESERVATION_REFINEMENT capability is not enabled.

• To specify the --type=container option for the docker inspect <container_name> command, the minimal supported Docker version has been updated from 1.0.0 to 1.8.0 since Docker supported --type=container for the docker inspect command starting from 1.8.0.

Upgrading from 1.2.x to 1.3.x

• The master will no longer allow 0.x agents to register. Interoperability between 1.1+ masters and 0.x agents has never been supported; however, it was not explicitly disallowed, either. Starting with this release of Mesos, registration attempts by 0.x agents will be ignored.

• Support for deprecated ACLs set_quotas and remove_quotas has been removed from the local authorizer. Before upgrading the Mesos binaries, consolidate the ACLs used under set_quotas and remove_quotes under their replacement ACL update_quotas. After consolidation of the ACLs, the binaries could be safely replaced.

• Support for deprecated ACL shutdown_frameworks has been removed from the local authorizer. Before upgrading the Mesos binaries, replace all instances of the ACL shutdown_frameworks with the newer ACL teardown_frameworks. After updating the ACLs, the binaries can be safely replaced.

• Support for multi-role frameworks deprecates the FrameworkInfo.role field in favor of FrameworkInfo.roles and the MULTI_ROLE capability. Frameworks using the new field can continue to use a single role.

• Support for multi-role frameworks means that the framework role field in the master and agent endpoints is deprecated in favor of roles. Any tooling parsing endpoint information and relying on the role field needs to be updated before multi-role frameworks can be safely run in the cluster.

• Implementors of allocator modules have to provide new implementation functionality to satisfy the MULTI_ROLE framework capability. Also, the interface has changed.

• New Agent flags authenticate_http_executors and executor_secret_key: Used to enable required HTTP executor authentication and set the key file used for generation and authentication of HTTP executor tokens. Note that enabling these flags after upgrade is disruptive to HTTP executors that were launched before the upgrade. For more information on the recommended upgrade procedure when enabling these flags, see the authentication documentation.

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents/schedulers can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 1.1.x to 1.2.x

• In Mesos 1.2.1, the master will no longer allow 0.x agents to register. Interoperability between 1.1+ masters and 0.x agents has never been supported; however, it was not explicitly disallowed, either. Starting with Mesos 1.2.1, registration attempts by 0.x agents will be ignored. NOTE: This applies only when upgrading to Mesos 1.2.1. Mesos 1.2.0 does not implement this behavior.

• New Agent flag http_heartbeat_interval: This flag sets a heartbeat interval for messages to be sent over persistent connections made against the agent HTTP API. Currently, this only applies to the LAUNCH_NESTED_CONTAINER_SESSION and ATTACH_CONTAINER_OUTPUT calls. (default: 30secs)

• New Agent flag image_provisioner_backend: Strategy for provisioning container rootfs from images, e.g., aufs, bind, copy, overlay.

• New Master flag max_unreachable_tasks_per_framework: Maximum number of unreachable tasks per framework to store in memory. (default: 1000)

• New Revive and Suppress v1 scheduler Calls: Revive or Suppress offers for a specified role. If role is unset, the call will revive/suppress offers for all of the roles the framework is subscribed to. (Especially for multi-role frameworks.)

• Mesos 1.2 modifies the ContainerLogger's prepare() method. The method now takes an additional argument for the user the logger should run a subprocess as. Please see MESOS-5856 for more information.

• Allocator module changes to support inactive frameworks, multi-role frameworks, and suppress/revive. See allocator.hpp for interface changes.

• New Authorizer module actions: LAUNCH_NESTED_CONTAINER, KILL_NESTED_CONTAINER, WAIT_NESTED_CONTAINER, LAUNCH_NESTED_CONTAINER_SESSION, ATTACH_CONTAINER_INPUT, ATTACH_CONTAINER_OUTPUT, VIEW_CONTAINER, and SET_LOG_LEVEL. See authorizer.proto for module interface changes, and acls.proto for corresponding LocalAuthorizer ACL changes.

• Renamed Authorizer module actions (and deprecated old aliases): REGISTER_FRAMEWORK, TEARDOWN_FRAMEWORK, RESERVE_RESOURCES, UNRESERVE_RESOURCES, CREATE_VOLUME, DESTROY_VOLUME, UPDATE_WEIGHT, GET_QUOTA. See authorizer.proto for interface changes.

• Removed slavePreLaunchDockerEnvironmentDecorator and slavePreLaunchDockerHook in favor of slavePreLaunchDockerTaskExecutorDecorator.

• New Agent v1 operator API calls: LAUNCH_NESTED_CONTAINER_SESSION, ATTACH_CONTAINER_INPUT, ATTACH_CONTAINER_OUTPUT for debugging into running containers (Mesos containerizer only).

• Deprecated recovered_frameworks in v1 GetFrameworks call. Now it will be empty.

• Deprecated orphan_executors in v1 GetExecutors call. Now it will be empty.

• Deprecated orphan_tasks in v1 GetTasks call. Now it will be empty.

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents/schedulers can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 1.0.x to 1.1.x

• Mesos 1.1 removes the ContainerLogger's recover() method. The ContainerLogger had an incomplete interface for a stateful implementation. This removes the incomplete parts to avoid adding tech debt in the containerizer. Please see MESOS-6371 for more information.

• Mesos 1.1 adds an offeredResources argument to the Allocator::updateAllocation() method. It is used to indicate the resources that the operations passed to updateAllocation() are applied to. MESOS-4431 (particularly /r/45961/) has more details on the motivation.

Upgrading from 0.28.x to 1.0.x

• Prior to Mesos 1.0, environment variables prefixed by SSL_ are used to control libprocess SSL support. However, it was found that those environment variables may collide with some libraries or programs (e.g., openssl, curl). From Mesos 1.0, SSL_* environment variables are deprecated in favor of the corresponding LIBPROCESS_SSL_* variables.

• Prior to Mesos 1.0, Mesos agent recursively changes the ownership of the persistent volumes every time they are mounted to a container. From Mesos 1.0, this behavior has been changed. Mesos agent will do a non-recursive change of ownership of the persistent volumes.

• Mesos 1.0 removed the camel cased protobuf fields in ContainerConfig (see include/mesos/slave/isolator.proto):
  • required ExecutorInfo executorInfo = 1;
  • optional TaskInfo taskInfo = 2;

• By default, executors will no longer inherit environment variables from the agent. The operator can still use the --executor_environment_variables flag on the agent to explicitly specify what environment variables the executors will get. Mesos generated environment variables (i.e., $MESOS_, $LIBPROCESS_) will not be affected. If $PATH is not specified for an executor, a default value /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin will be used.

• The allocator metric named allocator/event_queue_dispatches is now deprecated. The new name is allocator/mesos/event_queue_dispatches to better support metrics for alternative allocator implementations.

• The --docker_stop_timeout agent flag is deprecated.

• The ExecutorInfo.source field is deprecated in favor of ExecutorInfo.labels.

• Mesos 1.0 deprecates the 'slave' keyword in favor of 'agent' in a number of places
  • Deprecated flags with keyword 'slave' in favor of 'agent'.
  • Deprecated sandbox links with 'slave' keyword in the WebUI.
  • Deprecated slave subcommand for mesos-cli.

• Mesos 1.0 removes the default value for the agent's work_dir command-line flag. This flag is now required; the agent will exit immediately if it is not provided.

• Mesos 1.0 disables support for the master's registry_strict command-line flag. If this flag is specified, the master will exit immediately. Note that this flag was previously marked as experimental and not recommended for production use.

• Mesos 1.0 deprecates the use of plain text credential files in favor of JSON-formatted credential files.

• When a persistent volume is destroyed, Mesos will now remove any data that was stored on the volume from the filesystem of the appropriate agent. In prior versions of Mesos, destroying a volume would not delete data (this was a known missing feature that has now been implemented).

• Mesos 1.0 changes the HTTP status code of the following endpoints from 200 OK to 202 Accepted:
  • /reserve
  • /unreserve
  • /create-volumes
  • /destroy-volumes

• Added output_file field to CommandInfo.URI in Scheduler API and v1 Scheduler HTTP API.

• Changed Call and Event Type enums in scheduler.proto from required to optional for the purpose of backwards compatibility.

• Changed Call and Event Type enums in executor.proto from required to optional for the purpose of backwards compatibility.

• Added non-terminal task metadata to the container resource usage information.

• Deleted the /observe HTTP endpoint.

• The SetQuota and RemoveQuota ACLs have been deprecated. To replace these, a new ACL UpdateQuota have been introduced. In addition, a new ACL GetQuota have been added; these control which principals are allowed to query quota information for which roles. These changes affect the --acls flag for the local authorizer in the following ways:
  • The update_quotas ACL cannot be used in combination with either the set_quotas or remove_quotas ACL. The local authorizer will produce an error in such a case;
  • When upgrading a Mesos cluster that uses the set_quotas or remove_quotas ACLs, the operator should first upgrade the Mesos binaries. At this point, the deprecated ACLs will still be enforced. After the upgrade has been verified, the operator should replace deprecated values for set_quotas and remove_quotas with equivalent values for update_quotas;
  • If desired, the operator can use the get_quotas ACL after the upgrade to control which principals are allowed to query quota information.

• Mesos 1.0 contains a number of authorizer changes that particularly effect custom authorizer modules:
  • The authorizer interface has been refactored in order to decouple the ACL definition language from the interface. It additionally includes the option of retrieving ObjectApprover. An ObjectApprover can be used to synchronously check authorizations for a given object and is hence useful when authorizing a large number of objects and/or large objects (which need to be copied using request-based authorization). NOTE: This is a breaking change for authorizer modules.
  • Authorization-based HTTP endpoint filtering enables operators to restrict which parts of the cluster state a user is authorized to see. Consider for example the /state master endpoint: an operator can now authorize users to only see a subset of the running frameworks, tasks, or executors.
  • The subject and object fields in the authorization::Request protobuf message have been changed to be optional. If these fields are not set, the request should only be allowed for ACLs with ANY semantics. NOTE: This is a semantic change for authorizer modules.

• Namespace and header file of Allocator has been moved to be consistent with other packages.

• When a task is run as a particular user, the fetcher now fetches files as that user also. Note, this means that filesystem permissions for that user will be enforced when fetching local files.

• The --authenticate_http flag has been deprecated in favor of --authenticate_http_readwrite. Setting --authenticate_http_readwrite will now enable authentication for all endpoints which previously had authentication support. These happen to be the endpoints which allow modification of the cluster state, or "read-write" endpoints. Note that /logging/toggle, /profiler/start, /profiler/stop, /maintenance/schedule, /machine/up, and /machine/down previously did not have authentication support, but in 1.0 if either --authenticate_http or --authenticate_http_readwrite is set, those endpoints will now require authentication. A new flag has also been introduced, --authenticate_http_readonly, which enables authentication for endpoints which support authentication and do not allow modification of the state of the cluster, like /state or /flags.

• Mesos 1.0 introduces authorization support for several HTTP endpoints. Note that some of these endpoints are used by the web UI, and thus using the web UI in a cluster with authorization enabled will require that ACLs be set appropriately. Please refer to the authorization documentation for details.

• The endpoints with coarse-grained authorization enabled are:

  • /files/debug
  • /logging/toggle
  • /metrics/snapshot
  • /slave(id)/containers
  • /slave(id)/monitor/statistics

• If the defined ACLs used permissive: false, the listed HTTP endpoints will stop working unless ACLs for the get_endpoints actions are defined.

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 0.27.x to 0.28.x

• Mesos 0.28 only supports three decimal digits of precision for scalar resource values. For example, frameworks can reserve "0.001" CPUs but more fine-grained reservations (e.g., "0.0001" CPUs) are no longer supported (although they did not work reliably in prior versions of Mesos anyway). Internally, resource math is now done using a fixed-point format that supports three decimal digits of precision, and then converted to/from floating point for input and output, respectively. Frameworks that do their own resource math and manipulate fractional resources may observe differences in roundoff error and numerical precision.

• Mesos 0.28 changes the definitions of two ACLs used for authorization. The objects of the ReserveResources and CreateVolume ACLs have been changed to roles. In both cases, principals can now be authorized to perform these operations for particular roles. This means that by default, a framework or operator can reserve resources/create volumes for any role. To restrict this behavior, ACLs can be added to the master which authorize principals to reserve resources/create volumes for specified roles only. Previously, frameworks could only reserve resources for their own role; this behavior can be preserved by configuring the ReserveResources ACLs such that the framework's principal is only authorized to reserve for the framework's role. NOTE This renders existing ReserveResources and CreateVolume ACL definitions obsolete; if you are authorizing these operations, your ACL definitions should be updated.

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 0.26.x to 0.27.x

• Mesos 0.27 introduces the concept of implicit roles. In previous releases, configuring roles required specifying a static whitelist of valid role names on master startup (via the --roles flag). In Mesos 0.27, if --roles is omitted, any role name can be used; controlling which principals are allowed to register as which roles should be done using ACLs. The role whitelist functionality is still supported but is deprecated.

• The Allocator API has changed due to the introduction of implicit roles. Custom allocator implementations will need to be updated. See MESOS-4000 for more information.

• The executorLost callback in the Scheduler interface will now be called whenever the agent detects termination of a custom executor. This callback was never called in previous versions, so please make sure any framework schedulers can now safely handle this callback. Note that this callback may not be reliably delivered.

• The isolator prepare interface has been changed slightly. Instead of keeping adding parameters to the prepare interface, we decide to use a protobuf (ContainerConfig). Also, we renamed ContainerPrepareInfo to ContainerLaunchInfo to better capture the purpose of this struct. See MESOS-4240 and MESOS-4282 for more information. If you are an isolator module writer, you will have to adjust your isolator module according to the new interface and re-compile with 0.27.

• ACLs.shutdown_frameworks has been deprecated in favor of the new ACLs.teardown_frameworks. This affects the --acls master flag for the local authorizer.

• Reserved resources are now accounted for in the DRF role sorter. Previously unaccounted reservations will influence the weighted DRF sorter. If role weights were explicitly set, they may need to be adjusted in order to account for the reserved resources in the cluster.

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 0.25.x to 0.26.x

• The names of some TaskStatus::Reason enums have been changed. But the tag numbers remain unchanged, so it is backwards compatible. Frameworks using the new version might need to do some compile time adjustments:

  • REASON_MEM_LIMIT -> REASON_CONTAINER_LIMITATION_MEMORY
  • REASON_EXECUTOR_PREEMPTED -> REASON_CONTAINER_PREEMPTED

• The Credential protobuf has been changed. Credential field secret is now a string, it used to be bytes. This will affect framework developers and language bindings ought to update their generated protobuf with the new version. This fixes JSON based credentials file support.

• The /state endpoints on master and agent will no longer include data fields as part of the JSON models for ExecutorInfo and TaskInfo out of consideration for memory scalability (see MESOS-3794 and this email thread).
  • On master, the affected data field was originally found via frameworks[*].executors[*].data.
  • On agents, the affected data field was originally found via executors[*].tasks[*].data.

• The NetworkInfo protobuf has been changed. The fields protocol and ip_address are now deprecated. The new field ip_addresses subsumes the information provided by them.

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 0.24.x to 0.25.x

• The following endpoints will be deprecated in favor of new endpoints. Both versions will be available in 0.25 but the deprecated endpoints will be removed in a subsequent release.

  For master endpoints:

  • /state.json becomes /state
  • /tasks.json becomes /tasks

  For agent endpoints:

  • /state.json becomes /state
  • /monitor/statistics.json becomes /monitor/statistics

  For both master and agent:

  • /files/browse.json becomes /files/browse
  • /files/debug.json becomes /files/debug
  • /files/download.json becomes /files/download
  • /files/read.json becomes /files/read

• The C++/Java/Python scheduler bindings have been updated. In particular, the driver can make a suppressOffers() call to stop receiving offers (until reviveOffers() is called).

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 0.23.x to 0.24.x

• Support for live upgrading a driver based scheduler to HTTP based (experimental) scheduler has been added.

• Master now publishes its information in ZooKeeper in JSON (instead of protobuf). Make sure schedulers are linked against >= 0.23.0 libmesos before upgrading the master.

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 0.22.x to 0.23.x

• The 'stats.json' endpoints for masters and agents have been removed. Please use the 'metrics/snapshot' endpoints instead.

• The '/master/shutdown' endpoint is deprecated in favor of the new '/master/teardown' endpoint.

• In order to enable decorator modules to remove metadata (environment variables or labels), we changed the meaning of the return value for decorator hooks in Mesos 0.23.0. Please refer to the modules documentation for more details.

• Agent ping timeouts are now configurable on the master via --slave_ping_timeout and --max_slave_ping_timeouts. Agents should be upgraded to 0.23.x before changing these flags.

• A new scheduler driver API, acceptOffers, has been introduced. This is a more general version of the launchTasks API, which allows the scheduler to accept an offer and specify a list of operations (Offer.Operation) to perform using the resources in the offer. Currently, the supported operations include LAUNCH (launching tasks), RESERVE (making dynamic reservations), UNRESERVE (releasing dynamic reservations), CREATE (creating persistent volumes) and DESTROY (releasing persistent volumes). Similar to the launchTasks API, any unused resources will be considered declined, and the specified filters will be applied on all unused resources.

• The Resource protobuf has been extended to include more metadata for supporting persistence (DiskInfo), dynamic reservations (ReservationInfo) and oversubscription (RevocableInfo). You must not combine two Resource objects if they have different metadata.

In order to upgrade a running cluster:

1. Rebuild and install any modules so that upgraded masters/agents can use them.
2. Install the new master binaries and restart the masters.
3. Install the new agent binaries and restart the agents.
4. Upgrade the schedulers by linking the latest native library / jar / egg (if necessary).
5. Restart the schedulers.
6. Upgrade the executors by linking the latest native library / jar / egg (if necessary).

Upgrading from 0.21.x to 0.22.x

• Agent checkpoint flag has been removed as it will be enabled for all agents. Frameworks must still enable checkpointing during registration to take advantage of checkpointing their tasks.

• The stats.json endpoints for masters and agents have been deprecated. Please refer to the metrics/snapshot endpoint.

• The C++/Java/Python scheduler bindings have been updated. In particular, the driver can be constructed with an additional argument that specifies whether to use implicit driver acknowledgements. In statusUpdate, the TaskStatus now includes a UUID to make explicit acknowledgements possible.

• The Authentication API has changed slightly in this release to support additional authentication mechanisms. The change from 'string' to 'bytes' for AuthenticationStartMessage.data has no impact on C++ or the over-the-wire representation, so it only impacts pure language bindings for languages like Java and Python that use different types for UTF-8 strings vs. byte arrays.

  message AuthenticationStartMessage { required string mechanism = 1; optional bytes data = 2; }

• All Mesos arguments can now be passed using file:// to read them out of a file (either an absolute or relative path). The --credentials, --whitelist, and any flags that expect JSON backed arguments (such as --modules) behave as before, although support for just passing an absolute path for any JSON flags rather than file:// has been deprecated and will produce a warning (and the absolute path behavior will be removed in a future release).

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Install the new agent binaries and restart the agents.
3. Upgrade the schedulers:

• For Java schedulers, link the new native library against the new JAR. The JAR contains API above changes. A 0.21.0 JAR will work with a 0.22.0 libmesos. A 0.22.0 JAR will work with a 0.21.0 libmesos if explicit acks are not being used. 0.22.0 and 0.21.0 are inter-operable at the protocol level between the master and the scheduler.
• For Python schedulers, upgrade to use a 0.22.0 egg. If constructing MesosSchedulerDriverImpl with Credentials, your code must be updated to pass the implicitAcknowledgements argument before Credentials. You may run a 0.21.0 Python scheduler against a 0.22.0 master, and vice versa.

4. Restart the schedulers.
5. Upgrade the executors by linking the latest native library / jar / egg.

Upgrading from 0.20.x to 0.21.x

• Disabling agent checkpointing has been deprecated; the agent --checkpoint flag has been deprecated and will be removed in a future release.

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Install the new agent binaries and restart the agents.
3. Upgrade the schedulers by linking the latest native library (mesos jar upgrade not necessary).
4. Restart the schedulers.
5. Upgrade the executors by linking the latest native library and mesos jar (if necessary).

Upgrading from 0.19.x to 0.20.x.

• The Mesos API has been changed slightly in this release. The CommandInfo has been changed (see below), which makes launching a command more flexible. The 'value' field has been changed from required to optional. However, it will not cause any issue during the upgrade (since the existing schedulers always set this field).

    message CommandInfo {
      ...
      // There are two ways to specify the command:
      // 1) If 'shell == true', the command will be launched via shell
      //    (i.e., /bin/sh -c 'value'). The 'value' specified will be
      //    treated as the shell command. The 'arguments' will be ignored.
      // 2) If 'shell == false', the command will be launched by passing
      //    arguments to an executable. The 'value' specified will be
      //    treated as the filename of the executable. The 'arguments'
      //    will be treated as the arguments to the executable. This is
      //    similar to how POSIX exec families launch processes (i.e.,
      //    execlp(value, arguments(0), arguments(1), ...)).
      optional bool shell = 6 [default = true];
      optional string value = 3;
      repeated string arguments = 7;
      ...
    }
  

• The Python bindings are also changing in this release. There are now sub-modules which allow you to use either the interfaces and/or the native driver.

  • import mesos.native for the native drivers
  • import mesos.interface for the stub implementations and protobufs

  To ensure a smooth upgrade, we recommend to upgrade your python framework and executor first. You will be able to either import using the new configuration or the old. Replace the existing imports with something like the following:

  try: from mesos.native import MesosExecutorDriver, MesosSchedulerDriver from mesos.interface import Executor, Scheduler from mesos.interface import mesos_pb2 except ImportError: from mesos import Executor, MesosExecutorDriver, MesosSchedulerDriver, Scheduler import mesos_pb2

• If you're using a pure language binding, please ensure that it sends status update acknowledgements through the master before upgrading.

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Install the new agent binaries and restart the agents.
3. Upgrade the schedulers by linking the latest native library (install the latest mesos jar and python egg if necessary).
4. Restart the schedulers.
5. Upgrade the executors by linking the latest native library (install the latest mesos jar and python egg if necessary).

Upgrading from 0.18.x to 0.19.x.

• There are new required flags on the master (--work_dir and --quorum) to support the Registrar feature, which adds replicated state on the masters.

• No required upgrade ordering across components.

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Install the new agent binaries and restart the agents.
3. Upgrade the schedulers by linking the latest native library (mesos jar upgrade not necessary).
4. Restart the schedulers.
5. Upgrade the executors by linking the latest native library and mesos jar (if necessary).

Upgrading from 0.17.0 to 0.18.x.

• This upgrade requires a system reboot for agents that use Linux cgroups for isolation.

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Upgrade the schedulers by linking the latest native library and mesos jar (if necessary).
3. Restart the schedulers.
4. Install the new agent binaries then perform one of the following two steps, depending on if cgroups isolation is used:

• [no cgroups]
  • Restart the agents. The "--isolation" flag has changed and "process" has been deprecated in favor of "posix/cpu,posix/mem".
• [cgroups]
  • Change from a single mountpoint for all controllers to separate mountpoints for each controller, e.g., /sys/fs/cgroup/memory/ and /sys/fs/cgroup/cpu/.
  • The suggested configuration is to mount a tmpfs filesystem to /sys/fs/cgroup and to let the agent mount the required controllers. However, the agent will also use previously mounted controllers if they are appropriately mounted under "--cgroups_hierarchy".
  • It has been observed that unmounting and remounting of cgroups from the single to separate configuration is unreliable and a reboot into the new configuration is strongly advised. Restart the agents after reboot.
  • The "--cgroups_hierarchy" now defaults to "/sys/fs/cgroup". The "--cgroups_root" flag default remains "mesos".
  • The "--isolation" flag has changed and "cgroups" has been deprecated in favor of "cgroups/cpu,cgroups/mem".
  • The "--cgroup_subsystems" flag is no longer required and will be ignored.

5. Upgrade the executors by linking the latest native library and mesos jar (if necessary).

Upgrading from 0.16.0 to 0.17.0.

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Upgrade the schedulers by linking the latest native library and mesos jar (if necessary).
3. Restart the schedulers.
4. Install the new agent binaries and restart the agents.
5. Upgrade the executors by linking the latest native library and mesos jar (if necessary).

Upgrading from 0.15.0 to 0.16.0.

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Upgrade the schedulers by linking the latest native library and mesos jar (if necessary).
3. Restart the schedulers.
4. Install the new agent binaries and restart the agents.
5. Upgrade the executors by linking the latest native library and mesos jar (if necessary).

Upgrading from 0.14.0 to 0.15.0.

• Schedulers should implement the new reconcileTasks driver method.
• Schedulers should call the new MesosSchedulerDriver constructor that takes Credential to authenticate.
• --authentication=false (default) allows both authenticated and unauthenticated frameworks to register.

In order to upgrade a running cluster:

1. Install the new master binaries.
2. Restart the masters with --credentials pointing to credentials of the framework(s).
3. Install the new agent binaries and restart the agents.
4. Upgrade the executors by linking the latest native library and mesos jar (if necessary).
5. Upgrade the schedulers by linking the latest native library and mesos jar (if necessary).
6. Restart the schedulers. Restart the masters with --authentication=true.

NOTE: After the restart unauthenticated frameworks will not be allowed to register.

Upgrading from 0.13.0 to 0.14.0.

• /vars endpoint has been removed.

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Upgrade the executors by linking the latest native library and mesos jar (if necessary).
3. Install the new agent binaries.
4. Restart the agents after adding --checkpoint flag to enable checkpointing.
5. Upgrade the schedulers by linking the latest native library and mesos jar (if necessary).
6. Set FrameworkInfo.checkpoint in the scheduler if checkpointing is desired (recommended).
7. Restart the schedulers.
8. Restart the masters (to get rid of the cached FrameworkInfo).
9. Restart the agents (to get rid of the cached FrameworkInfo).

Upgrading from 0.12.0 to 0.13.0.

• cgroups_hierarchy_root agent flag is renamed as cgroups_hierarchy

In order to upgrade a running cluster:

1. Install the new master binaries and restart the masters.
2. Upgrade the schedulers by linking the latest native library and mesos jar (if necessary).
3. Restart the schedulers.
4. Install the new agent binaries.
5. Restart the agents.
6. Upgrade the executors by linking the latest native library and mesos jar (if necessary).

Upgrading from 0.11.0 to 0.12.0.

• If you are a framework developer, you will want to examine the new 'source' field in the ExecutorInfo protobuf. This will allow you to take further advantage of the resource monitoring.

In order to upgrade a running cluster:

1. Install the new agent binaries and restart the agents.
2. Install the new master binaries and restart the masters.

Downgrade Mesos

This document serves as a guide for users who wish to downgrade from an existing Mesos cluster to a previous version. This usually happens when rolling back from problematic upgrades. Mesos provides compatibility between any 1.x and 1.y versions of masters/agents as long as new features are not used. Since Mesos 1.8, we introduced a check for minimum capabilities on the master. If a backwards incompatible feature is used, a corresponding minimum capability entry will be persisted to the registry. If an old master (that does not possess the capability) tries to recover from the registry (e.g. when rolling back), an error message will be printed containing the missing capabilities. This document lists the detailed information regarding these minimum capabilities and remediation for downgrade errors.

List of Master Minimum Capabilities

Logging

Mesos handles the logs of each Mesos component differently depending on the degree of control Mesos has over the source code of the component.

Roughly, these categories are:

• Internal - Master and Agent.
• Containers - Executors and Tasks.
• External - Components launched outside of Mesos, like Frameworks and ZooKeeper. These are expected to implement their own logging solution.

Internal

The Mesos Master and Agent use the Google's logging library. For information regarding the command-line options used to configure this library, see the configuration documentation. Google logging options that are not explicitly mentioned there can be configured via environment variables.

Both Master and Agent also expose a /logging/toggle HTTP endpoint which temporarily toggles verbose logging:

POST <ip:port>/logging/toggle?level=[1|2|3]&duration=VALUE

The effect is analogous to setting the GLOG_v environment variable prior to starting the Master/Agent, except the logging level will revert to the original level after the given duration.

Containers

For background, see the containerizer documentation.

Mesos does not assume any structured logging for entities running inside containers. Instead, Mesos will store the stdout and stderr of containers into plain files ("stdout" and "stderr") located inside the sandbox.

In some cases, the default Container logger behavior of Mesos is not ideal:

• Logging may not be standardized across containers.
• Logs are not easily aggregated.
• Log file sizes are not managed. Given enough time, the "stdout" and "stderr" files can fill up the Agent's disk.

ContainerLogger Module

The ContainerLogger module was introduced in Mesos 0.27.0 and aims to address the shortcomings of the default logging behavior for containers. The module can be used to change how Mesos redirects the stdout and stderr of containers.

The interface for a ContainerLogger can be found here.

Mesos comes with two ContainerLogger modules:

• The SandboxContainerLogger implements the existing logging behavior as a ContainerLogger. This is the default behavior.
• The LogrotateContainerLogger addresses the problem of unbounded log file sizes.

LogrotateContainerLogger

The LogrotateContainerLogger constrains the total size of a container's stdout and stderr files. The module does this by rotating log files based on the parameters to the module. When a log file reaches its specified maximum size, it is renamed by appending a .N to the end of the filename, where N increments each rotation. Older log files are deleted when the specified maximum number of files is reached.

Invoking the module

The LogrotateContainerLogger can be loaded by specifying the library liblogrotate_container_logger.so in the --modules flag when starting the Agent and by setting the --container_logger Agent flag to org_apache_mesos_LogrotateContainerLogger.

Module parameters

  Defaults to 10 MB.  Minimum size of 1 (memory) page, usually around 4 KB.
</td>

/path/to/stdout {
  [logrotate_stdout_options]
  size [max_stdout_size]
}

  Defaults to <code>CONTAINER_LOGGER_</code>.
</td>

  Defaults to <code>/usr/local/libexec/mesos</code>.
</td>

How it works

1. Every time a container starts up, the LogrotateContainerLogger starts up companion subprocesses of the mesos-logrotate-logger binary.
2. The module instructs Mesos to redirect the container's stdout/stderr to the mesos-logrotate-logger.
3. As the container outputs to stdout/stderr, mesos-logrotate-logger will pipe the output into the "stdout"/"stderr" files. As the files grow, mesos-logrotate-logger will call logrotate to keep the files strictly under the configured maximum size.
4. When the container exits, mesos-logrotate-logger will finish logging before exiting as well.

The LogrotateContainerLogger is designed to be resilient across Agent failover. If the Agent process dies, any instances of mesos-logrotate-logger will continue to run.

Writing a Custom ContainerLogger

For basics on module writing, see the modules documentation.

There are several caveats to consider when designing a new ContainerLogger:

• Logging by the ContainerLogger should be resilient to Agent failover. If the Agent process dies (which includes the ContainerLogger module), logging should continue. This is usually achieved by using subprocesses.
• When containers shut down, the ContainerLogger is not explicitly notified. Instead, encountering EOF in the container's stdout/stderr signifies that the container has exited. This provides a stronger guarantee that the ContainerLogger has seen all the logs before exiting itself.
• The ContainerLogger should not assume that containers have been launched with any specific ContainerLogger. The Agent may be restarted with a different ContainerLogger.
• Each containerizer running on an Agent uses its own instance of the ContainerLogger. This means more than one ContainerLogger may be running in a single Agent. However, each Agent will only run a single type of ContainerLogger.

Mesos Observability Metrics

This document describes the observability metrics provided by Mesos master and agent nodes. This document also provides some initial guidance on which metrics you should monitor to detect abnormal situations in your cluster.

Overview

Mesos master and agent nodes report a set of statistics and metrics that enable cluster operators to monitor resource usage and detect abnormal situations early. The information reported by Mesos includes details about available resources, used resources, registered frameworks, active agents, and task state. You can use this information to create automated alerts and to plot different metrics over time inside a monitoring dashboard.

Metric information is not persisted to disk at either master or agent nodes, which means that metrics will be reset when masters and agents are restarted. Similarly, if the current leading master fails and a new leading master is elected, metrics at the new master will be reset.

Metric Types

Mesos provides two different kinds of metrics: counters and gauges.

Counters keep track of discrete events and are monotonically increasing. The value of a metric of this type is always a natural number. Examples include the number of failed tasks and the number of agent registrations. For some metrics of this type, the rate of change is often more useful than the value itself.

Gauges represent an instantaneous sample of some magnitude. Examples include the amount of used memory in the cluster and the number of connected agents. For some metrics of this type, it is often useful to determine whether the value is above or below a threshold for a sustained period of time.

The tables in this document indicate the type of each available metric.

Master Nodes

Metrics from each master node are available via the /metrics/snapshot master endpoint. The response is a JSON object that contains metrics names and values as key-value pairs.

Observability metrics

This section lists all available metrics from Mesos master nodes grouped by category.

Resources

The following metrics provide information about the total resources available in the cluster and their current usage. High resource usage for sustained periods of time may indicate that you need to add capacity to your cluster or that a framework is misbehaving.

Master

The following metrics provide information about whether a master is currently elected and how long it has been running. A cluster with no elected master for sustained periods of time indicates a malfunctioning cluster. This points to either leadership election issues (so check the connection to ZooKeeper) or a flapping Master process. A low uptime value indicates that the master has restarted recently.

System

The following metrics provide information about the resources available on this master node and their current usage. High resource usage in a master node for sustained periods of time may degrade the performance of the cluster.

Agents

The following metrics provide information about agent events, agent counts, and agent states. A low number of active agents may indicate that agents are unhealthy or that they are not able to connect to the elected master.

Frameworks

The following metrics provide information about the registered frameworks in the cluster. No active or connected frameworks may indicate that a scheduler is not registered or that it is misbehaving.

The following metrics are added for each framework which registers with the master, in order to provide detailed information about the behavior of the framework. The framework name is percent-encoded before creating these metrics; the actual name can be recovered by percent-decoding.

Tasks

The following metrics provide information about active and terminated tasks. A high rate of lost tasks may indicate that there is a problem with the cluster. The task states listed here match those of the task state machine.

Operations

The following metrics provide information about offer operations on the master.

Below, OPERATION_TYPE refers to any one of reserve, unreserve, create, destroy, grow_volume, shrink_volume, create_disk or destroy_disk.

NOTE: The counter for terminal operation states can over-count over time. In particular if an agent contained unacknowledged terminal status updates when it was marked gone or marked unreachable, these operations will be double-counted as both their original state and OPERATION_GONE/OPERATION_UNREACHABLE.

Messages

The following metrics provide information about messages between the master and the agents and between the framework and the executors. A high rate of dropped messages may indicate that there is a problem with the network.

Event queue

The following metrics provide information about different types of events in the event queue.

Registrar

The following metrics provide information about read and write latency to the agent registrar.

Replicated log

The following metrics provide information about the replicated log underneath the registrar, which is the persistent store for masters.

Allocator

The following metrics provide information about performance and resource allocations in the allocator.

Basic Alerts

This section lists some examples of basic alerts that you can use to detect abnormal situations in a cluster.

master/uptime_secs is low

The master has restarted.

master/uptime_secs < 60 for sustained periods of time

The cluster has a flapping master node.

master/tasks_lost is increasing rapidly

Tasks in the cluster are disappearing. Possible causes include hardware failures, bugs in one of the frameworks, or bugs in Mesos.

master/slaves_active is low

Agents are having trouble connecting to the master.

master/cpus_percent > 0.9 for sustained periods of time

Cluster CPU utilization is close to capacity.

master/mem_percent > 0.9 for sustained periods of time

Cluster memory utilization is close to capacity.

master/elected is 0 for sustained periods of time

No master is currently elected.

Agent Nodes

Metrics from each agent node are available via the /metrics/snapshot agent endpoint. The response is a JSON object that contains metrics names and values as key-value pairs.

Observability Metrics

This section lists all available metrics from Mesos agent nodes grouped by category.

Resources

The following metrics provide information about the total resources available in the agent and their current usage.

Agent

The following metrics provide information about whether an agent is currently registered with a master and for how long it has been running.

System

The following metrics provide information about the agent system.

Executors

The following metrics provide information about the executor instances running on the agent.

Tasks

The following metrics provide information about active and terminated tasks.

Messages

The following metrics provide information about messages between the agents and the master it is registered with.

Containerizers

The following metrics provide information about both Mesos and Docker containerizers.

Resource Providers

The following metrics provide information about ongoing and completed operations that apply to resources provided by a resource provider with the given type and name. In the following metrics, the operation placeholder refers to the name of a particular operation type, which is described in the list of supported operation types.

Supported Operation Types

Since the supported operation types may vary among different resource providers, the following is a comprehensive list of operation types and the corresponding resource providers that support them. Note that the name column is for the operation placeholder in the above metrics.

For example, cluster operators can monitor the number of successful CREATE_VOLUME operations that are applied to the resource provider with type org.apache.mesos.rp.local.storage and name lvm through the resource_providers/org.apache.mesos.rp.local.storage.lvm/operations/create_disk/finished metric.

CSI Plugins

Storage resource providers in Mesos are backed by CSI plugins running in standalone containers. To monitor the health of these CSI plugins for a storage resource provider with type and name, the following metrics provide information about plugin terminations and ongoing and completed CSI calls made to the plugin.

The new CLI

The new Mesos Command Line Interface provides one executable Python 3 script to run all default commands and additional custom plugins.

Two of the subcommands available allow you to debug running containers:

• mesos task exec, to run a command in a running task's container.
• mesos task attach, to attach your local terminal to a running task and stream its input/output.

Building the CLI

For now, the Mesos CLI is still under development and not built as part of a standard Mesos distribution.

However, the CLI can be built using Autotools and Cmake options. If necessary, check the options described in the linked pages to set Python 3 before starting a build.

The result of this build will be a mesos binary that can be executed.

Using the CLI

Using the CLI without building Mesos is also possible. To do so, activate the CLI virtual environment by following the steps described below:

$ cd src/python/cli_new/
$ PYTHON=python3 ./bootstrap
$ source activate
$ mesos

Calling mesos will then run the CLI and calling mesos-cli-tests will run the integration tests.

Configuring the CLI

The CLI uses a configuration file to know where the masters of the cluster are as well as list any plugins that should be used in addition to the default ones provided.

The configuation file, located by default at ~/.mesos/config.toml, looks like this:

# The `plugins` array lists the absolute paths of the
# plugins you want to add to the CLI.
plugins = [
  "</absolute/path/to/plugin-1/directory>",
  "</absolute/path/to/plugin-2/directory>"
]

# The `master` field is either composed of an `address` field
# or a `zookeeper` field, but not both. For example:
[master]
  address = "10.10.0.30:5050"
  # The `zookeeper` field has an `addresses` array and a `path` field.
  # [master.zookeeper]
  #   addresses = [
  #     "10.10.0.31:5050",
  #     "10.10.0.32:5050",
  #     "10.10.0.33:5050"
  #   ]
  #   path = "/mesos"

Operational Guide

Using a process supervisor

Mesos uses a "fail-fast" approach to error handling: if a serious error occurs, Mesos will typically exit rather than trying to continue running in a possibly erroneous state. For example, when Mesos is configured for high availability, the leading master will abort itself when it discovers it has been partitioned away from the Zookeeper quorum. This is a safety precaution to ensure the previous leader doesn't continue communicating in an unsafe state.

To ensure that such failures are handled appropriately, production deployments of Mesos typically use a process supervisor (such as systemd or supervisord) to detect when Mesos processes exit. The supervisor can be configured to restart the failed process automatically and/or to notify the cluster operator to investigate the situation.

Changing the master quorum

The master leverages a Paxos-based replicated log as its storage backend (--registry=replicated_log is the only storage backend currently supported). Each master participates in the ensemble as a log replica. The --quorum flag determines a majority of the masters.

The following table shows the tolerance to master failures for each quorum size:

It is recommended to run with 3 or 5 masters, when desiring high availability.

NOTE

When configuring the quorum, it is essential to ensure that there are only so many masters running as specified in the table above. If additional masters are running, this violates the quorum and the log may be corrupted! As a result, it is recommended to gate the running of the master process with something that enforces a static whitelist of the master hosts. See MESOS-1546 for adding a safety whitelist within Mesos itself.

For online reconfiguration of the log, see: MESOS-683.

Increasing the quorum size

As the size of a cluster grows, it may be desired to increase the quorum size for additional fault tolerance.

The following steps indicate how to increment the quorum size, using 3 -> 5 masters as an example (quorum size 2 -> 3):

1. Initially, 3 masters are running with --quorum=2
2. Restart the original 3 masters with --quorum=3
3. Start 2 additional masters with --quorum=3

To increase the quorum by N, repeat this process to increment the quorum size N times.

NOTE: Currently, moving out of a single master setup requires wiping the replicated log state and starting fresh. This will wipe all persistent data (e.g., agents, maintenance information, quota information, etc). To move from 1 master to 3 masters:

1. Stop the standalone master.
2. Remove the replicated log data (replicated_log under the --work_dir).
3. Start the original master and two new masters with --quorum=2

Decreasing the quorum size

The following steps indicate how to decrement the quorum size, using 5 -> 3 masters as an example (quorum size 3 -> 2):

1. Initially, 5 masters are running with --quorum=3
2. Remove 2 masters from the cluster, ensure they will not be restarted (see NOTE section above). Now 3 masters are running with --quorum=3
3. Restart the 3 masters with --quorum=2

To decrease the quorum by N, repeat this process to decrement the quorum size N times.

Replacing a master

Please see the NOTE section above. So long as the failed master is guaranteed to not re-join the ensemble, it is safe to start a new master with an empty log and allow it to catch up.

External access for Mesos master

If the default IP (or the command line arg --ip) is an internal IP, then external entities such as framework schedulers will be unable to reach the master. To address that scenario, an externally accessible IP:port can be setup via the --advertise_ip and --advertise_port command line arguments of mesos-master. If configured, external entities such as framework schedulers interact with the advertise_ip:advertise_port from where the request needs to be proxied to the internal IP:port on which the Mesos master is listening.

HTTP requests to non-leading master

HTTP requests to some master endpoints (e.g., /state, /machine/down) can only be answered by the leading master. Such requests made to a non-leading master will result in either a 307 Temporary Redirect (with the location of the leading master) or 503 Service Unavailable (if the master does not know who the current leader is).

Mesos Fetcher

Mesos 0.23.0 introduced experimental support for the Mesos fetcher cache.

In this context we loosely regard the term "downloading" as to include copying from local file systems.

What is the Mesos fetcher?

The Mesos fetcher is a mechanism to download resources into the sandbox directory of a task in preparation of running the task. As part of a TaskInfo message, the framework ordering the task's execution provides a list of CommandInfo::URI protobuf values, which becomes the input to the Mesos fetcher.

The Mesos fetcher can copy files from a local filesytem and it also natively supports the HTTP, HTTPS, FTP and FTPS protocols. If the requested URI is based on some other protocol, then the fetcher tries to utilise a local Hadoop client and hence supports any protocol supported by the Hadoop client, e.g., HDFS, S3. See the agent configuration documentation for how to configure the agent with a path to the Hadoop client.

By default, each requested URI is downloaded directly into the sandbox directory and repeated requests for the same URI leads to downloading another copy of the same resource. Alternatively, the fetcher can be instructed to cache URI downloads in a dedicated directory for reuse by subsequent downloads.

The Mesos fetcher mechanism comprises of these two parts:

1. The agent-internal Fetcher Process (in terms of libprocess) that controls and coordinates all fetch actions. Every agent instance has exactly one internal fetcher instance that is used by every kind of containerizer.

2. The external program mesos-fetcher that is invoked by the former. It performs all network and disk operations except file deletions and file size queries for cache-internal bookkeeping. It is run as an external OS process in order to shield the agent process from I/O-related hazards. It takes instructions in form of an environment variable containing a JSON object with detailed fetch action descriptions.

The fetch procedure

Frameworks launch tasks by calling the scheduler driver method launchTasks(), passing CommandInfo protobuf structures as arguments. This type of structure specifies (among other things) a command and a list of URIs that need to be "fetched" into the sandbox directory on the agent node as a precondition for task execution. Hence, when the agent receives a request to launch a task, it calls upon its fetcher, first, to provision the specified resources into the sandbox directory. If fetching fails, the task is not started and the reported task status is TASK_FAILED.

All URIs requested for a given task are fetched sequentially in a single invocation of mesos-fetcher. Here, avoiding download concurrency reduces the risk of bandwidth issues somewhat. However, multiple fetch operations can be active concurrently due to multiple task launch requests.

The URI protobuf structure

Before mesos-fetcher is started, the specific fetch actions to be performed for each URI are determined based on the following protobuf structure. (See include/mesos/mesos.proto for more details.)

message CommandInfo {
  message URI {
    required string value = 1;
    optional bool executable = 2;
    optional bool extract = 3 [default = true];
    optional bool cache = 4;
    optional string output_file = 5;
  }
  ...
  optional string user = 5;
}

The field "value" contains the URI.

If the "executable" field is "true", the "extract" field is ignored and has no effect.

If the "cache" field is true, the fetcher cache is to be used for the URI.

If the "output_file" field is set, the fetcher will use that name for the copy stored in the sandbox directory. "output_file" may contain a directory component, in which case the path described must be a relative path.

Specifying a user name

The framework may pass along a user name that becomes a fetch parameter. This causes its executors and tasks to run under a specific user. However, if the "user" field in the CommandInfo structure is specified, it takes precedence for the affected task.

If a user name is specified either way, the fetcher first validates that it is in fact a valid user name on the agent. If it is not, fetching fails right here. Otherwise, the sandbox directory is assigned to the specified user as owner (using chown) at the end of the fetch procedure, before task execution begins.

The user name in play has an important effect on caching. Caching is managed on a per-user base, i.e. the combination of user name and "uri" uniquely identifies a cacheable fetch result. If no user name has been specified, this counts for the cache as a separate user, too. Thus cache files for each valid user are segregated from all others, including those without a specified user.

This means that the exact same URI will be downloaded and cached multiple times if different users are indicated.

Executable fetch results

By default, fetched files are not executable.

If the field "executable" is set to "true", the fetch result will be changed to be executable (by "chmod") for every user. This happens at the end of the fetch procedure, in the sandbox directory only. It does not affect any cache file.

Archive extraction

If the "extract" field is "true", which is the default, then files with a recognized extension that hints at packed or compressed archives are unpacked in the sandbox directory. These file extensions are recognized:

• .tar, .tar.gz, .tar.bz2, .tar.xz
• .gz, .tgz, .tbz2, .txz, .zip

In case the cache is bypassed, both the archive and the unpacked results will be found together in the sandbox. In case a cache file is unpacked, only the extraction result will be found in the sandbox.

The "output_file" field is useful here for cases where the URI ends with query parameters, since these will otherwise end up in the file copied to the sandbox and will subsequently fail to be recognized as archives.

Bypassing the cache

By default, the URI field "cache" is not present. If this is the case or its value is "false" the fetcher downloads directly into the sandbox directory.

The same also happens dynamically as a fallback strategy if anything goes wrong when preparing a fetch operation that involves the cache. In this case, a warning message is logged. Possible fallback conditions are:

• The server offering the URI does not respond or reports an error.
• The URI's download size could not be determined.
• There is not enough space in the cache, even after attempting to evict files.

Fetching through the cache

If the URI's "cache" field has the value "true", then the fetcher cache is in effect. If a URI is encountered for the first time (for the same user), it is first downloaded into the cache, then copied to the sandbox directory from there. If the same URI is encountered again, and a corresponding cache file is resident in the cache or still en route into the cache, then downloading is omitted and the fetcher proceeds directly to copying from the cache. Competing requests for the same URI simply wait upon completion of the first request that occurs. Thus every URI is downloaded at most once (per user) as long as it is cached.

Every cache file stays resident for an unspecified amount of time and can be removed at the fetcher's discretion at any moment, except while it is in direct use:

• It is still being downloaded by this fetch procedure.
• It is still being downloaded by a concurrent fetch procedure for a different task.
• It is being copied or extracted from the cache.

Once a cache file has been removed, the related URI will thereafter be treated as described above for the first encounter.

Unfortunately, there is no mechanism to refresh a cache entry in the current experimental version of the fetcher cache. A future feature may force updates based on checksum queries to the URI.

Recommended practice for now:

The framework should start using a fresh unique URI whenever the resource's content has changed.

Determining resource sizes

Before downloading a resource to the cache, the fetcher first determines the size of the expected resource. It uses these methods depending on the nature of the URI.

• Local file sizes are probed with systems calls (that follow symbolic links).
• HTTP/HTTPS URIs are queried for the "content-length" field in the header. This is performed by curl. The reported asset size must be greater than zero or the URI is deemed invalid.
• FTP/FTPS is not supported at the time of writing.
• Everything else is queried by the local HDFS client.

If any of this reports an error, the fetcher then falls back on bypassing the cache as described above.

WARNING: Only URIs for which download sizes can be queried up front and for which accurate sizes are reported reliably are eligible for any fetcher cache involvement. If actual cache file sizes exceed the physical capacity of the cache directory in any way, all further agent behavior is completely unspecified. Do not use any cache feature with any URI for which you have any doubts!

To mitigate this problem, cache files that have been found to be larger than expected are deleted immediately after downloading and delivering the requested content to the sandbox. Thus exceeding total capacity at least does not accumulate over subsequent fetcher runs.

If you know for sure that size aberrations are within certain limits you can specify a cache directory size that is sufficiently smaller than your actual physical volume and fetching should work.

In case of cache files that are smaller then expected, the cache will dynamically adjust its own bookkeeping according to actual sizes.

Cache eviction

After determining the prospective size of a cache file and before downloading it, the cache attempts to ensure that at least as much space as is needed for this file is available and can be written into. If this is immediately the case, the requested amount of space is simply marked as reserved. Otherwise, missing space is freed up by "cache eviction". This means that the cache removes files at its own discretion until the given space target is met or exceeded.

The eviction process fails if too many files are in use and therefore not evictable or if the cache is simply too small. Either way, the fetcher then falls back on bypassing the cache for the given URI as described above.

If multiple evictions happen concurrently, each of them is pursuing its own separate space goals. However, leftover freed up space from one effort is automatically awarded to others.

HTTP and SOCKS proxy settings

Sometimes it is desirable to use a proxy to download the file. The Mesos fetcher uses libcurl internally for downloading content from HTTP/HTTPS/FTP/FTPS servers, and libcurl can use a proxy automatically if certain environment variables are set.

The respective environment variable name is [protocol]_proxy, where protocol can be one of socks4, socks5, http, https.

For example, the value of the http_proxy environment variable would be used as the proxy for fetching http contents, while https_proxy would be used for fetching https contents. Pay attention that these variable names must be entirely in lower case.

The value of the proxy variable is of the format [protocol://][user:password@]machine[:port], where protocol can be one of socks4, socks5, http, https.

FTP/FTPS requests with a proxy also make use of an HTTP/HTTPS proxy. Even though in general this constrains the available FTP protocol operations, everything the fetcher uses is supported.

Your proxy settings can be placed in /etc/default/mesos-slave. Here is an example:

export http_proxy=https://proxy.example.com:3128
export https_proxy=https://proxy.example.com:3128

The fetcher will pick up these environment variable settings since the utility program mesos-fetcher which it employs is a child of mesos-agent.

For more details, please check the libcurl manual.

Agent flags

It is highly recommended to set these flags explicitly to values other than their defaults or to not use the fetcher cache in production.

• "fetcher_cache_size", default value: enough for testing.
• "fetcher_cache_dir", default value: somewhere inside the directory specified by the "work_dir" flag, which is OK for testing.

Recommended practice:

• Use a separate volume as fetcher cache. Do not specify a directory as fetcher cache directory that competes with any other contributor for the underlying volume's space.
• Set the cache directory size flag of the agent to less than your actual cache volume's physical size. Use a safety margin, especially if you do not know for sure if all frameworks are going to be compliant.

Ultimate remedy:

You can disable the fetcher cache entirely on each agent by setting its "fetcher_cache_size" flag to zero bytes.

Future Features

The following features would be relatively easy to implement additionally.

• Perform cache updates based on resource check sums. For example, query the md5 field in HTTP headers to determine when a resource at a URL has changed.
• Respect HTTP cache-control directives.
• Enable caching for ftp/ftps.
• Use symbolic links or bind mounts to project cached resources into the sandbox, read-only.
• Have a choice whether to copy the extracted archive into the sandbox.
• Have a choice whether to delete the archive after extraction bypassing the cache.
• Make the segregation of cache files by user optional.
• Extract content while downloading when bypassing the cache.
• Prefetch resources for subsequent tasks. This can happen concurrently with running the present task, right after fetching its own resources.

Implementation Details

The Mesos Fetcher Cache Internals describes how the fetcher cache is implemented.

title: Apache Mesos - Domains and Regions layout: documentation

Regions and Fault Domains

Starting with Mesos 1.5, it is possible to place Mesos masters and agents into domains, which are logical groups of machines that share some characteristics.

Currently, fault domains are the only supported type of domains, which are groups of machines with similar failure characteristics.

A fault domain is a 2 level hierarchy of regions and zones. The mapping from fault domains to physical infrastructure is up to the operator to configure, although it is recommended that machines in the same zones have low latency to each other.

In cloud environments, regions and zones can be mapped to the "region" and "availability zone" concepts exposed by most cloud providers, respectively. In on-premise deployments, regions and zones can be mapped to data centers and racks, respectively.

Schedulers may prefer to place network-intensive workloads in the same domain, as this may improve performance. Conversely, a single failure that affects a host in a domain may be more likely to affect other hosts in the same domain; hence, schedulers may prefer to place workloads that require high availability in multiple domains. For example, all the hosts in a single rack might lose power or network connectivity simultaneously.

The --domain flag can be used to specify the fault domain of a master or agent node. The value of this flag must be a file path or a JSON dictionary with the key fault_domain and subkeys region and zone mapping to arbitrary strings:

mesos-master --domain='{"fault_domain": {"region": {"name":"eu"}, "zone": { "name":"rack1"}}}'

mesos-agent  --domain='{"fault_domain": {"region": {"name":"eu"}, "zone": {"name":"rack2"}}}'

Frameworks can learn about the domain of an agent by inspecting the domain field in the received offer, which contains a DomainInfo that has the same structure as the JSON dictionary above.

Constraints

When configuring fault domains for the masters and agents, the following constraints must be obeyed:

• If a mesos master is not configured with a domain, it will reject connection attempts from agents with a domain.

  This is done because the master is not able to determine whether or not the agent would be remote in this case.

• Agents with no configured domain are assumed to be in the same domain as the master.

  If this behaviour isn't desired, the --require_agent_domain flag on the master can be used to enforce that domains are configured on all agents by having the master reject all registration attempts by agents without a configured domain.

• If one master is configured with a domain, all other masters must be in the same "region" to avoid cross-region quorum writes. It is recommended to put them in different zones within that region for high availability.

• The default DRF resource allocator will only offer resources from agents in the same region as the master. To receive offers from all regions, a framework must set the REGION_AWARE capability bit in its FrameworkInfo.

Example

A short example will serve to illustrate these concepts. WayForward Technologies runs a successful website that allows users to purchase things that they want to have.

To do this, it owns a data center in San Francisco, in which it runs a number of custom Mesos frameworks. All agents within the data center are configured with the same region sf, and the individual racks inside the data center are used as zones.

The three mesos masters are placed in different server racks in the data center, which gives them enough isolation to withstand events like a whole rack losing power or network connectivity but still have low-enough latency for quorum writes.

One of the provided services is a real-time view of the company's inventory. The framework providing this service is placing all of its tasks in the same zone as the database server, to take advantage of the high-speed, low-latency link so it can always display the latest results.

During peak hours, it might happen that the computing power required to operate the website exceeds the capacity of the data center. To avoid unnecessary hardware purchases, WayForward Technologies contracted with a third-party cloud provider TPC. The machines from this provider are placed in a different region tpc, and the zones are configured to correspond to the availability zones provided by TPC. All relevant frameworks are updated with the REGION_AWARE bit in their FrameworkInfo and their scheduling logic is updated so that they can schedule tasks in the cloud if required.

Non-region aware frameworks will now only receive offers from agents within the data center, where the master nodes reside. Region-aware frameworks are supposed to know when and if they should place their tasks in the data center or with the cloud provider.

Performance Profiling

This document over time will be home to various guides on how to use various profiling tools to do performance analysis of Mesos.

Flamescope

Flamescope is a visualization tool for exploring different time ranges as flamegraphs. In order to use the tool, you first need to obtain stack traces, here's how to obtain a 60 second recording of the mesos master process at 100 hertz using Linux perf:

$ sudo perf record --freq=100 --no-inherit --call-graph dwarf -p <mesos-master-pid> -- sleep 60
$ sudo perf script --header | c++filt > mesos-master.stacks
$ gzip mesos-master.stacks

If you'd like to solicit help in analyzing the performance data, upload the mesos-master.stacks.gz to a publicly accessible location and file with dev@mesos.apache.org for analysis, or send the file over slack to the #performance channel.

Alternatively, to do the analysis yourself, place mesos-master.stacks into the examples folder of a flamescope git checkout.

Memory Profiling with Mesos and Jemalloc

On Linux systems, Mesos is able to leverage the memory-profiling capabilities of the jemalloc general-purpose allocator to provide powerful debugging tools for investigating memory-related issues.

These include detailed real-time statistics of the current memory usage, as well as information about the location and frequency of individual allocations.

This generally works by having libprocess detect at runtime whether the current process is using jemalloc as its memory allocator, and if so enable a number of HTTP endpoints described below that allow operators to generate the desired data at runtime.

Requirements

A prerequisite for memory profiling is a suitable allocator. Currently only jemalloc is supported, which can be connected via one of the following ways.

The recommended method is to specify the --enable-jemalloc-allocator compile-time flag, which causes the mesos-master and mesos-agent binaries to be statically linked against a bundled version of jemalloc that will be compiled with the correct compile-time flags.

Alternatively and analogous to other bundled dependencies of Mesos, it is of course also possible to use a suitable custom version of jemalloc with the --with-jemalloc=</path-to-jemalloc> flag.

NOTE: Suitable here means that jemalloc should have been built with the --enable-stats and --enable-prof flags, and that the string prof:true;prof_active:false is part of the malloc configuration. The latter condition can be satisfied either at configuration or at run-time, see the section on MALLOC_CONF below.

The third way is to use the LD_PRELOAD mechanism to preload a libjemalloc.so shared library that is present on the system at runtime. The MemoryProfiler class in libprocess will automatically detect this and enable its memory profiling support.

The generated profile dumps will be written to a random directory under TMPDIR if set, otherwise in a subdirectory of /tmp.

Finally, note that since jemalloc was designed to be used in highly concurrent allocation scenarios, it can improve performance over the default system allocator. In this case, it can be beneficial to build Mesos with jemalloc even if there is no intention to use the memory profiling functionality.

Usage

There are two independent sets of data that can be collected from jemalloc: memory statistics and heap profiling information.

Using any of the endpoints described below requires the jemalloc allocator and starting the mesos-agent or mesos-master binary with the option --memory_profiling=true (or setting the environment variable LIBPROCESS_MEMORY_PROFILING=true for other binaries using libprocess).

Memory Statistics

The /statistics endpoint returns exact statistics about the memory usage in JSON format, for example the number of bytes currently allocated and the size distribution of these allocations.

It takes no parameters and will return the results in JSON format:

http://example.org:5050/memory-profiler/statistics

Be aware that the returned JSON is quite large, so when accessing this endpoint from a terminal, it is advisable to redirect the results into a file.

Heap Profiling

The profiling done by jemalloc works by sampling from the calls to malloc() according to a configured probability distribution, and storing stack traces for the sampled calls in a separate memory area. These can then be dumped into files on the filesystem, so-called heap profiles.

To start a profiling run one would access the /start endpoint:

http://example.org:5050/memory-profiler/start?duration=5mins

followed by downloading one of the generated files described below after the duration has elapsed. The remaining time of the current profiling run can be verified via the /state endpoint:

http://example.org:5050/memory-profiler/state

Since profiling information is stored process-global by jemalloc, only a single concurrent profiling run is allowed. Additionally, only the results of the most recently finished run are stored on disk.

The profile collection can also be stopped early with the /stop endpoint:

http://example.org:5050/memory-profiler/stop

To analyze the generated profiling data, the results are offered in three different formats.

Raw profile

http://example.org:5050/memory-profiler/download/raw

This returns a file in a plain text format containing the raw backtraces collected, i.e., lists of memory addresses. It can be interactively analyzed and rendered using the jeprof tool provided by the jemalloc project. For more information on this file format, check out the official jemalloc documentation.

Symbolized profile

http://example.org:5050/memory-profiler/download/text

This is similar to the raw format above, except that jeprof is called on the host machine to attempt to read symbol information from the current binary and replace raw memory addresses in the profile by human-readable symbol names.

Usage of this endpoint requires that jeprof is present on the host machine and on the PATH, and no useful information will be generated unless the binary contains symbol information.

Call graph

http://example.org:5050/memory-profiler/download/graph

This endpoint returns an image in SVG format that shows a graphical representation of the samples backtraces.

Usage of this endpoint requires that jeprof and dot are present on the host machine and on the PATH of mesos, and no useful information will be generated unless the binary contains symbol information.

Overview

Which of these is needed will depend on the circumstances of the application deployment and of the bug that is investigated.

For example, the call graph presents information in a visual, immediately useful form, but is difficult to filter and post-process if non-default output options are desired.

On the other hand, in many debian-like environments symbol information is by default stripped from binaries to save space and shipped in separate packages. In such an environment, if it is not permitted to install additional packages on the host running Mesos, one would store the raw profiles and enrich them with symbol information locally.

Jeprof Installation

As described above, the /download/text and /download/graph endpoints require the jeprof program installed on the host system. Where possible, it is recommended to install jeprof through the system package manager, where it is usually packaged alongside with jemalloc itself.

Alternatively, a copy of the script can be found under 3rdparty/jemalloc-5.0.1/bin/jeprof in the build directory, or can be downloaded directly from the internet using a command like:

$ curl https://raw.githubusercontent.com/jemalloc/jemalloc/dev/bin/jeprof.in | sed s/@jemalloc_version@/5.0.1/ >jeprof

Note that jeprof is just a perl script that post-processes the raw profiles. It has no connection to the jemalloc library besides being distributed in the same package. In particular, it is generally not required to have matching versions of jemalloc and jeprof.

If jeprof is installed manually, one also needs to take care to install the necessary dependencies. In particular, this include the perl interpreter to execute the script itself and the dot binary to generate graph files.

Command-line Usage

In some circumstances, it might be desired to automate the downloading of heap profiles by writing a simple script. A simple example for how this might look like this:

#!/bin/bash

SECONDS=600
HOST=example.org:5050

curl ${HOST}/memory-profiler/start?duration=${SECONDS}
sleep $((${SECONDS} + 1))
wget ${HOST}/memory-profiler/download/raw

A more sophisticated script would additionally store the id value returned by the call to /start and pass it as a paremter to /download, to ensure that a new run was not started in the meantime.

Using the MALLOC_CONF Interface

The jemalloc allocator provides a native interface to control the memory profiling behaviour. The usual way to provide settings through this interface is by setting the environment variable MALLOC_CONF.

NOTE: If libprocess detects that memory profiling was started through MALLOC_CONF, it will reject starting a profiling run of its own to avoid interference.

The MALLOC_CONF interface provides a number of options that are not exposed by libprocess, like generating heap profiles automatically after a certain amount of memory has been allocated, or whenever memory usage reaches a new high-water mark. The full list of settings is described on the jemalloc man page.

On the other hand, features like starting and stopping the profiling at runtime or getting the information provided by the /statistics endpoint can not be achieved through the MALLOC_CONF interface.

For example, to create a dump automatically for every 1 GiB worth of recorded allocations, one might use the configuration:

MALLOC_CONF="prof:true,prof_prefix:/path/to/folder,lg_prof_interval=20"

To debug memory allocations during early startup, profiling can be activated before accessing the /start endpoint:

MALLOC_CONF="prof:true,prof_active:true"

Mesos Attributes & Resources

Mesos has two basic methods to describe the agents that comprise a cluster. One of these is managed by the Mesos master, the other is simply passed onwards to the frameworks using the cluster.

Types

The types of values that are supported by Attributes and Resources in Mesos are scalar, ranges, sets and text.

The following are the definitions of these types:

scalar : floatValue

floatValue : ( intValue ( "." intValue )? ) | ...

intValue : [0-9]+

range : "[" rangeValue ( "," rangeValue )* "]"

rangeValue : scalar "-" scalar

set : "{" text ( "," text )* "}"

text : [a-zA-Z0-9_/.-]

Attributes

Attributes are key-value pairs (where value is optional) that Mesos passes along when it sends offers to frameworks. An attribute value supports three different types: scalar, range or text.

attributes : attribute ( ";" attribute )*

attribute : text ":" ( scalar | range | text )

Note that setting multiple attributes corresponding to the same key is highly discouraged (and might be disallowed in future), as this complicates attribute- based filtering of offers, both on schedulers side and on the Mesos side.

Resources

Mesos can manage three different types of resources: scalars, ranges, and sets. These are used to represent the different resources that a Mesos agent has to offer. For example, a scalar resource type could be used to represent the amount of memory on an agent. Scalar resources are represented using floating point numbers to allow fractional values to be specified (e.g., "1.5 CPUs"). Mesos only supports three decimal digits of precision for scalar resources (e.g., reserving "1.5123 CPUs" is considered equivalent to reserving "1.512 CPUs"). For GPUs, Mesos only supports whole number values.

Resources can be specified either with a JSON array or a semicolon-delimited string of key-value pairs. If, after examining the examples below, you have questions about the format of the JSON, inspect the Resource protobuf message definition in include/mesos/mesos.proto.

As JSON:

[
  {
    "name": "<resource_name>",
    "type": "SCALAR",
    "scalar": {
      "value": <resource_value>
    }
  },
  {
    "name": "<resource_name>",
    "type": "RANGES",
    "ranges": {
      "range": [
        {
          "begin": <range_beginning>,
          "end": <range_ending>
        },
        ...
      ]
    }
  },
  {
    "name": "<resource_name>",
    "type": "SET",
    "set": {
      "item": [
        "<first_item>",
        ...
      ]
    },
    "role": "<role_name>"
  },
  ...
]

As a list of key-value pairs:

resources : resource ( ";" resource )*

resource : key ":" ( scalar | range | set )

key : text ( "(" resourceRole ")" )?

resourceRole : text | "*"

Note that resourceRole must be a valid role name; see the roles documentation for details.

Predefined Uses & Conventions

There are several kinds of resources that have predefined behavior:

• cpus
• gpus
• disk
• mem
• ports

Note that disk and mem resources are specified in megabytes. The master's user interface will convert resource values into a more human-readable format: for example, the value 15000 will be displayed as 14.65GB.

An agent without cpus and mem resources will not have its resources advertised to any frameworks.

Examples

By default, Mesos will try to autodetect the resources available at the local machine when mesos-agent starts up. Alternatively, you can explicitly configure which resources an agent should make available.

Here are some examples of how to configure the resources at a Mesos agent:

--resources='cpus:24;gpus:2;mem:24576;disk:409600;ports:[21000-24000,30000-34000];bugs(debug_role):{a,b,c}'

--resources='[{"name":"cpus","type":"SCALAR","scalar":{"value":24}},{"name":"gpus","type":"SCALAR","scalar":{"value":2}},{"name":"mem","type":"SCALAR","scalar":{"value":24576}},{"name":"disk","type":"SCALAR","scalar":{"value":409600}},{"name":"ports","type":"RANGES","ranges":{"range":[{"begin":21000,"end":24000},{"begin":30000,"end":34000}]}},{"name":"bugs","type":"SET","set":{"item":["a","b","c"]},"role":"debug_role"}]'

Or given a file resources.txt containing the following:

[
  {
    "name": "cpus",
    "type": "SCALAR",
    "scalar": {
      "value": 24
    }
  },
  {
    "name": "gpus",
    "type": "SCALAR",
    "scalar": {
      "value": 2
    }
  },
  {
    "name": "mem",
    "type": "SCALAR",
    "scalar": {
      "value": 24576
    }
  },
  {
    "name": "disk",
    "type": "SCALAR",
    "scalar": {
      "value": 409600
    }
  },
  {
    "name": "ports",
    "type": "RANGES",
    "ranges": {
      "range": [
        {
          "begin": 21000,
          "end": 24000
        },
        {
          "begin": 30000,
          "end": 34000
        }
      ]
    }
  },
  {
    "name": "bugs",
    "type": "SET",
    "set": {
      "item": [
        "a",
        "b",
        "c"
      ]
    },
    "role": "debug_role"
  }
]

You can do:

$ path/to/mesos-agent --resources=file:///path/to/resources.txt ...

In this case, we have five resources of three different types: scalars, a range, and a set. There are scalars called cpus, gpus, mem and disk, a range called ports, and a set called bugs. bugs is assigned to the role debug_role, while the other resources do not specify a role and are thus assigned to the default role.

Note: the "default role" can be set by the --default_role flag.

• scalar called cpus, with the value 24
• scalar called gpus, with the value 2
• scalar called mem, with the value 24576
• scalar called disk, with the value 409600
• range called ports, with values 21000 through 24000 and 30000 through 34000 (inclusive)
• set called bugs, with the values a, b and c, assigned to the role debug_role

To configure the attributes of a Mesos agent, you can use the --attributes command-line flag of mesos-agent:

--attributes='rack:abc;zone:west;os:centos5;level:10;keys:[1000-1500]'

That will result in configuring the following five attributes:

• rack with text value abc
• zone with text value west
• os with text value centos5
• level with scalar value 10
• keys with range value 1000 through 1500 (inclusive)

title: Apache Mesos - Roles layout: documentation

Roles

Many modern host-level operating systems (e.g. Linux, BSDs, etc) support multiple users. Similarly, Mesos is a multi-user cluster management system, with the expectation of a single Mesos cluster managing an organization's resources and servicing the organization's users.

As such, Mesos has to address a number of requirements related to resource management:

• Fair sharing of the resources amongst users
• Providing resource guarantees to users (e.g. quota, priorities, isolation)
• Providing accurate resource accounting
  • How many resources are allocated / utilized / etc?
  • Per-user accounting

In Mesos, we refer to these "users" as roles. More precisely, a role within Mesos refers to a resource consumer within the cluster. This resource consumer could represent a user within an organization, but it could also represent a team, a group, a service, a framework, etc.

Schedulers subscribe to one or more roles in order to receive resources and schedule work on behalf of the resource consumer(s) they are servicing.

Some examples of resource allocation guarantees that Mesos provides:

• Guaranteeing that a role is allocated a specified amount of resources (via quota).
• Ensuring that some (or all) of the resources on a particular agent are allocated to a particular role (via reservations).
• Ensuring that resources are fairly shared between roles (via DRF).
• Expressing that some roles should receive a higher relative share of the cluster (via weights).

Roles and access control

There are two ways to control which roles a framework is allowed to subscribe to. First, ACLs can be used to specify which framework principals can subscribe to which roles. For more information, see the authorization documentation.

Second, a role whitelist can be configured by passing the --roles flag to the Mesos master at startup. This flag specifies a comma-separated list of role names. If the whitelist is specified, only roles that appear in the whitelist can be used. To change the whitelist, the Mesos master must be restarted. Note that in a high-availability deployment of Mesos, you should take care to ensure that all Mesos masters are configured with the same whitelist.

In Mesos 0.26 and earlier, you should typically configure both ACLs and the whitelist, because in these versions of Mesos, any role that does not appear in the whitelist cannot be used.

In Mesos 0.27, this behavior has changed: if --roles is not specified, the whitelist permits any role name to be used. Hence, in Mesos 0.27, the recommended practice is to only use ACLs to define which roles can be used; the --roles command-line flag is deprecated.

Associating frameworks with roles

A framework specifies which roles it would like to subscribe to when it subscribes with the master. This is done via the roles field in FrameworkInfo. A framework can also change which roles it is subscribed to by reregistering with an updated FrameworkInfo.

As a user, you can typically specify which role(s) a framework will subscribe to when you start the framework. How to do this depends on the user interface of the framework you're using. For example, a single user scheduler might take a --mesos_role command-line flag and a multi-user scheduler might take a --mesos-roles command-line flag or sync with the organization's LDAP system to automatically adjust which roles it is subscribed to as the organization's structure changes.

Subscribing to multiple roles

As noted above, a framework can subscribe to multiple roles simultaneously. Frameworks that want to do this must opt-in to the MULTI_ROLE capability.

When a framework is offered resources, those resources are associated with exactly one of the roles it has subscribed to; the framework can determine which role an offer is for by consulting the allocation_info.role field in the Offer or the allocation_info.role field in each offered Resource (in the current implementation, all the resources in a single Offer will be allocated to the same role).

Multiple frameworks in the same role

Multiple frameworks can be subscribed to the same role. This can be useful: for example, one framework can create a persistent volume and write data to it. Once the task that writes data to the persistent volume has finished, the volume will be offered to other frameworks subscribed to the same role; this might give a second ("consumer") framework the opportunity to launch a task that reads the data produced by the first ("producer") framework.

However, configuring multiple frameworks to use the same role should be done with caution, because all the frameworks will have access to any resources that have been reserved for that role. For example, if a framework stores sensitive information on a persistent volume, that volume might be offered to a different framework subscribed to the same role. Similarly, if one framework creates a persistent volume, another framework subscribed to the same role might "steal" the volume and use it to launch a task of its own. In general, multiple frameworks sharing the same role should be prepared to collaborate with one another to ensure that role-specific resources are used appropriately.

Associating resources with roles

A resource is assigned to a role using a reservation. Resources can either be reserved statically (when the agent that hosts the resource is started) or dynamically: frameworks and operators can specify that a certain resource should subsequently be reserved for use by a given role. For more information, see the reservation documentation.

Default role

The role named * is special. Unreserved resources are currently represented as having the special * role (the idea being that * matches any role). By default, all the resources at an agent node are unreserved (this can be changed via the --default_role command-line flag when starting the agent).

In addition, when a framework registers without providing a FrameworkInfo.role, it is assigned to the * role. In Mesos 1.3, frameworks should use the FrameworkInfo.roles field, which does not assign a default of *, but frameworks can still specify * explicitly if desired. Frameworks and operators cannot make reservations to the * role.

Invalid role names

A role name must be a valid directory name, so it cannot:

• Be an empty string
• Be . or ..
• Start with -
• Contain any slash, backspace, or whitespace character

Roles and resource allocation

By default, the Mesos master uses weighted Dominant Resource Fairness (wDRF) to allocate resources. In particular, this implementation of wDRF first identifies which role is furthest below its fair share of the role's dominant resource. Each of the frameworks subscribed to that role are then offered additional resources in turn.

The resource allocation process can be customized by assigning weights to roles: a role with a weight of 2 will be allocated twice the fair share of a role with a weight of 1. By default, every role has a weight of 1. Weights can be configured using the /weights operator endpoint, or else using the deprecated --weights command-line flag when starting the Mesos master.

Roles and quota

In order to guarantee that a role is allocated a specific amount of resources, quota can be specified via the /quota endpoint.

The resource allocator will first attempt to satisfy the quota requirements, before fairly sharing the remaining resources. For more information, see the quota documentation.

Role vs. Principal

A principal identifies an entity that interacts with Mesos; principals are similar to user names. For example, frameworks supply a principal when they register with the Mesos master, and operators provide a principal when using the operator HTTP endpoints. An entity may be required to authenticate with its principal in order to prove its identity, and the principal may be used to authorize actions performed by an entity, such as resource reservation and persistent volume creation/destruction.

Roles, on the other hand, are used exclusively for resource allocation, as covered above.

title: Apache Mesos - Weights layout: documentation

Weights

In Mesos, weights can be used to control the relative share of cluster resources that is offered to different roles.

In Mesos 0.28 and earlier, weights can only be configured by specifying the --weights command-line flag when starting the Mesos master. If a role does not have a weight specified in the --weights flag, then the default value (1.0) will be used. Weights cannot be changed without updating the flag and restarting all Mesos masters.

Mesos 1.0 contains a /weights operator endpoint that allows weights to be changed at runtime. The --weights command-line flag is deprecated.

Operator HTTP Endpoint

The master /weights HTTP endpoint enables operators to configure weights. The endpoint currently offers a REST-like interface and supports the following operations:

• Updating weights with PUT.
• Querying the currently set weights with GET.

The endpoint can optionally use authentication and authorization. See the authentication guide for details.

Update

The operator can update the weights by sending an HTTP PUT request to the /weights endpoint.

An example request to the /weights endpoint could look like this (using the JSON file below):

$ curl -d @weights.json -X PUT http://<master-ip>:<port>/weights

For example, to set a weight of 2.0 for role1 and set a weight of 3.5 for role2, the operator can use the following weights.json:

    [
      {
        "role": "role1",
        "weight": 2.0
      },
      {
        "role": "role2",
        "weight": 3.5
      }
    ]

If the master is configured with an explicit role whitelist, the request is only valid if all specified roles exist in the role whitelist.

Weights are now persisted in the registry on cluster bootstrap and after any updates. Once the weights are persisted in the registry, any Mesos master that subsequently starts with --weights still specified will emit a warning and use the registry value instead.

The operator will receive one of the following HTTP response codes:

• 200 OK: Success (the update request was successful).
• 400 BadRequest: Invalid arguments (e.g., invalid JSON, non-positive weights).
• 401 Unauthorized: Unauthenticated request.
• 403 Forbidden: Unauthorized request.

Query

The operator can query the configured weights by sending an HTTP GET request to the /weights endpoint.

$ curl -X GET http://<master-ip>:<port>/weights

The response message body includes a JSON representation of the current configured weights, for example:

    [
      {
        "role": "role2",
        "weight": 3.5
      },
      {
        "role": "role1",
        "weight": 2.0
      }
    ]

The operator will receive one of the following HTTP response codes:

• 200 OK: Success.
• 401 Unauthorized: Unauthenticated request.

title: Apache Mesos - Quota layout: documentation

Quota

When multiple users are sharing a cluster, the operator may want to set limits on how many resources each user can use. Quota addresses this need and allows operators to set these limits on a per-role basis.

• Supported Resources
• Setting Quotas
• Viewing Quotas
• Deprecated: Quota Guarantees
• Implementation Notes

Supported Resources

The following resources have quota support:

• cpus
• mem
• disk
• gpus
• any custom resource of type SCALAR

The following resources do not have quota support:

• ports
• any custom resource of type RANGES or SET

Updating Quotas

By default, every role has no resource limits. To modify the resource limits for one or more roles, the v1 API UPDATE_QUOTA call is used. Note that this call applies the update in an all-or-nothing manner, so that if one of the role's quota updates is invalid or unauthorized, the entire request will not go through.

Example:

curl --request POST \
     --url http://<master-ip>:<master-port>/api/v1/ \
     --header 'Content-Type: application/json' \
     --data '{
               "type": "UPDATE_QUOTA",
               "update_quota": {
                 "force": false,
                 "quota_configs": [
                   {
                     "role": "dev",
                     "limits": {
                       "cpus": { "value": 10 },
                       "mem":  { "value": 2048 },
                       "disk": { "value": 4096 }
                     }
                   },
                   {
                     "role": "test",
                     "limits": {
                       "cpus": { "value": 1 },
                       "mem":  { "value": 256 },
                       "disk": { "value": 512 }
                     }
                   }
                 ]
               }
             }'

• Note that the request will be denied if the current quota consumption is above the provided limit. This check can be overriden by setting force to true.
• Note that the master will attempt to rescind a sufficient number of offers to ensure that the role cannot exceed its limits.

Viewing Quotas

Web UI

The 'Roles' tab in the web ui displays resource accounting information for all known roles. This includes the configured quota and the quota consumption.

API

There are several endpoints for viewing quota related information.

The v1 API GET_QUOTA call will return the quota configuration:

$ curl --request POST \
     --url http://<master-ip>:<master-port>/api/v1/ \
     --header 'Content-Type: application/json' \
     --header 'Accept: application/json' \
     --data '{ "type": "GET_QUOTA" }'

Response:

{
  "type": "GET_QUOTA",
  "get_quota": {
    "status": {
      "infos": [
        {
          "configs" : [
            {
              "role": "dev",
              "limits": {
                "cpus": { "value": 10.0 },
                "mem":  { "value": 2048.0 },
                "disk": { "value": 4096.0 }
              }
            },
            {
              "role": "test",
              "limits": {
                "cpus": { "value": 1.0 },
                "mem":  { "value": 256.0 },
                "disk": { "value": 512.0 }
              }
            }
          ]
        }
      ]
    }
  }
}

To also view the quota consumption, use the /roles endpoint:

$ curl http://<master-ip>:<master-port>/roles

Response

{
  "roles": [
    {
      "name": "dev",
      "weight": 1.0,
      "quota":
      {
        "role": "dev",
        "limit": {
          "cpus": 10.0,
          "mem":  2048.0,
          "disk": 4096.0
        },
        "consumed": {
          "cpus": 2.0,
          "mem":  1024.0,
          "disk": 2048.0
        }
      },
      "allocated": {
        "cpus": 2.0,
        "mem":  1024.0,
        "disk": 2048.0
      },
      "offered": {},
      "reserved": {
        "cpus": 2.0,
        "mem":  1024.0,
        "disk": 2048.0
      },
      "frameworks": []
    }
  ]
}

Quota Consumption

A role's quota consumption consists of its allocations and reservations. In other words, even if reservations are not allocated, they are included in the quota consumption. Offered resources are not charged against quota.

Metrics

The following metric keys are exposed for quota:

• allocator/mesos/quota/roles/<role>/resources/<resource>/guarantee
• allocator/mesos/quota/roles/<role>/resources/<resource>/limit
• A quota consumption metric will be added via MESOS-9123.

Deprecated: Quota Guarantees

Prior to Mesos 1.9, the quota related APIs only exposed quota "guarantees" which ensured a minimum amount of resources would be available to a role. Setting guarantees also set implicit quota limits. In Mesos 1.9+, quota limits are now exposed directly per the above documentation.

Quota guarantees are now deprecated in favor of using only quota limits. Enforcement of quota guarantees required that Mesos holds back enough resources to meet all of the unsatisfied quota guarantees. Since Mesos is moving towards an optimistic offer model (to improve multi-role / multi- scheduler scalability, see MESOS-1607), it will become no longer possible to enforce quota guarantees by holding back resources. In such a model, quota limits are simple to enforce, but quota guarantees would require a complex "effective limit" propagation model to leave space for unsatisfied guarantees.

For these reasons, quota guarantees, while still functional in Mesos 1.9, are now deprecated. A combination of limits and priority based preemption will be simpler in an optimistic offer model.

For documentation on quota guarantees, please see the previous documentation: https://github.com/apache/mesos/blob/1.8.0/docs/quota.md

Implementation Notes

• Quota is not supported on nested roles (e.g. eng/prod).
• During failover, in order to correctly enforce limits, the allocator will be paused and will not issue offers until at least 80% agents re-register or 10 minutes elapses. These parameters will be made configurable: MESOS-4073
• Quota is SUPPORTED for the default * role now MESOS-3938.

title: Apache Mesos - Reservation layout: documentation

Reservation

Mesos provides mechanisms to reserve resources in specific slaves. The concept was first introduced with static reservation in 0.14.0 which enabled operators to specify the reserved resources on slave startup. This was extended with dynamic reservation in 0.23.0 which enabled operators and authorized frameworks to dynamically reserve resources in the cluster.

In both types of reservations, resources are reserved for a role.

Static Reservation

An operator can configure a slave with resources reserved for a role. The reserved resources are specified via the --resources flag. For example, suppose we have 12 CPUs and 6144 MB of RAM available on a slave and that we want to reserve 8 CPUs and 4096 MB of RAM for the ads role. We start the slave like so:

    $ mesos-slave \
      --master=<ip>:<port> \
      --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096"

We now have 8 CPUs and 4096 MB of RAM reserved for ads on this slave.

CAVEAT: In order to modify a static reservation, the operator must drain and restart the slave with the new configuration specified in the --resources flag.

It's often more convenient to specify the total resources available on the slave as unreserved via the --resources flag and manage reservations dynamically (see below) via the master HTTP endpoints. However static reservation provides a way for the operator to more deterministically control the reservations (roles, amount, principals) before the agent is exposed to the master and frameworks. One use case is for the operator to dedicate entire agents for specific roles.

Dynamic Reservation

As mentioned in Static Reservation, specifying the reserved resources via the --resources flag makes the reservation static. That is, statically reserved resources cannot be reserved for another role nor be unreserved. Dynamic reservation enables operators and authorized frameworks to reserve and unreserve resources after slave-startup.

• Offer::Operation::Reserve and Offer::Operation::Unreserve messages are available for frameworks to send back via the acceptOffers API as a response to a resource offer.
• /reserve and /unreserve HTTP endpoints allow operators to manage dynamic reservations through the master.

In the following sections, we will walk through examples of each of the interfaces described above.

If two dynamic reservations are made for the same role at a single slave (using the same labels, if any; see below), the reservations will be combined by adding together the resources reserved by each request. This will result in a single reserved resource at the slave. Similarly, "partial" unreserve operations are allowed: an unreserve operation can release some but not all of the resources at a slave that have been reserved for a role. In this case, the unreserved resources will be subtracted from the previous reservation and any remaining resources will still be reserved.

Dynamic reservations cannot be unreserved if they are still being used by a running task or if a persistent volume has been created using the reserved resources. In the latter case, the volume should be destroyed before unreserving the resources.

Authorization

By default, frameworks and operators are authorized to reserve resources for any role and to unreserve dynamically reserved resources. Authorization allows this behavior to be limited so that resources can only be reserved for particular roles, and only particular resources can be unreserved. For these operations to be authorized, the framework or operator should provide a principal to identify itself. To use authorization with reserve/unreserve operations, the Mesos master must be configured with the appropriate ACLs. For more information, see the authorization documentation.

Similarly, agents by default can register with the master with resources that are statically reserved for arbitrary roles. With authorization, the master can be configured to use the reserve_resources ACL to check that the agent's principal is allowed to statically reserve resources for specific roles.

Reservation Labels

Dynamic reservations can optionally include a list of labels, which are arbitrary key-value pairs. Labels can be used to associate arbitrary metadata with a resource reservation. For example, frameworks can use labels to identify the intended purpose for a portion of the resources that have been reserved at a given slave. Note that two reservations with different labels will not be combined together into a single reservation, even if the reservations are at the same slave and use the same role.

Reservation Refinement

Hierarhical roles such as eng/backend enable the delegation of resources down a hierarchy, and reservation refinement is the mechanism with which reservations are delegated down the hierarchy. For example, a reservation (static or dynamic) for eng can be refined to eng/backend. When such a reservation is unreserved, they are returned to the previous owner. In this case it would be returned to eng. Reservation refinements can also "skip" levels. For example, eng can be refined directly to eng/backend/db. Again, unreserving such a reservation is returned to its previous owner eng.

NOTE: Frameworks need to enable the RESERVATION_REFINEMENT capability in order to be offered, and to create refined reservations

Listing Reservations

Information about the reserved resources at each slave in the cluster can be found by querying the /slaves master endpoint (under the reserved_resources_full key).

The same information can also be found in the /state endpoint on the agent (under the reserved_resources_full key). The agent endpoint is useful to confirm if a reservation has been propagated to the agent (which can fail in the event of network partition or master/agent restarts).

Examples

Framework Scheduler API

Offer::Operation::Reserve (without RESERVATION_REFINEMENT)

A framework can reserve resources through the resource offer cycle. The reservation role must match the offer's allocation role. Suppose we receive a resource offer with 12 CPUs and 6144 MB of RAM unreserved, allocated to role "engineering".

    {
      "allocation_info": { "role": "engineering" },
      "id": <offer_id>,
      "framework_id": <framework_id>,
      "slave_id": <slave_id>,
      "hostname": <hostname>,
      "resources": [
        {
          "allocation_info": { "role": "engineering" },
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 12 },
          "role": "*",
        },
        {
          "allocation_info": { "role": "engineering" },
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 6144 },
          "role": "*",
        }
      ]
    }

We can reserve 8 CPUs and 4096 MB of RAM by sending the following Offer::Operation message. Offer::Operation::Reserve has a resources field which we specify with the resources to be reserved. We must explicitly set the resources' role field to the offer's allocation role. The required value of the principal field depends on whether or not the framework provided a principal when it registered with the master. If a principal was provided, then the resources' principal field must be equal to the framework's principal. If no principal was provided during registration, then the resources' principal field can take any value, or can be left unset. Note that the principal field determines the "reserver principal" when authorization is enabled, even if authentication is disabled.

    {
      "type": Offer::Operation::RESERVE,
      "reserve": {
        "resources": [
          {
            "allocation_info": { "role": "engineering" },
            "name": "cpus",
            "type": "SCALAR",
            "scalar": { "value": 8 },
            "role": "engineering",
            "reservation": {
              "principal": <framework_principal>
            }
          },
          {
            "allocation_info": { "role": "engineering" },
            "name": "mem",
            "type": "SCALAR",
            "scalar": { "value": 4096 },
            "role": "engineering",
            "reservation": {
              "principal": <framework_principal>
            }
          }
        ]
      }
    }

If the reservation is successful, a subsequent resource offer will contain the following reserved resources:

    {
      "allocation_info": { "role": "engineering" },
      "id": <offer_id>,
      "framework_id": <framework_id>,
      "slave_id": <slave_id>,
      "hostname": <hostname>,
      "resources": [
        {
          "allocation_info": { "role": "engineering" },
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 8 },
          "role": "engineering",
          "reservation": {
            "principal": <framework_principal>
          }
        },
        {
          "allocation_info": { "role": "engineering" },
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 4096 },
          "role": "engineering",
          "reservation": {
            "principal": <framework_principal>
          }
        },
      ]
    }

Offer::Operation::Unreserve (without RESERVATION_REFINEMENT)

A framework can unreserve resources through the resource offer cycle. In Offer::Operation::Reserve, we reserved 8 CPUs and 4096 MB of RAM on a particular slave for one of our subscribed roles (e.g. "engineering"). The master will continue to only offer these reserved resources to the reservation's role. Suppose we would like to unreserve these resources. First, we receive a resource offer (copy/pasted from above):

    {
      "allocation_info": { "role": "engineering" },
      "id": <offer_id>,
      "framework_id": <framework_id>,
      "slave_id": <slave_id>,
      "hostname": <hostname>,
      "resources": [
        {
          "allocation_info": { "role": "engineering" },
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 8 },
          "role": "engineering",
          "reservation": {
            "principal": <framework_principal>
          }
        },
        {
          "allocation_info": { "role": "engineering" },
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 4096 },
          "role": "engineering",
          "reservation": {
            "principal": <framework_principal>
          }
        },
      ]
    }

We can unreserve the 8 CPUs and 4096 MB of RAM by sending the following Offer::Operation message. Offer::Operation::Unreserve has a resources field which we can use to specify the resources to be unreserved.

    {
      "type": Offer::Operation::UNRESERVE,
      "unreserve": {
        "resources": [
          {
            "allocation_info": { "role": "engineering" },
            "name": "cpus",
            "type": "SCALAR",
            "scalar": { "value": 8 },
            "role": "engineering",
            "reservation": {
              "principal": <framework_principal>
            }
          },
          {
            "allocation_info": { "role": "engineering" },
            "name": "mem",
            "type": "SCALAR",
            "scalar": { "value": 4096 },
            "role": "engineering",
            "reservation": {
              "principal": <framework_principal>
            }
          }
        ]
      }
    }

The unreserved resources may now be offered to other frameworks.

Offer::Operation::Reserve (with RESERVATION_REFINEMENT)

A framework that wants to create a refined reservation needs to enable the RESERVATION_REFINEMENT capability. Doing so will allow the framework to use the reservations field in the Resource message in order to push a refined reservation.

Since reserved resources are offered to any of the child roles under the role for which they are reserved for, they can get allocated to say, "engineering/backend" while being reserved for "engineering". It can then be refined to be reserved for "engineering/backend".

Note that the refined reservation role must match the offer's allocation role.

Suppose we receive a resource offer with 12 CPUs and 6144 MB of RAM reserved to "engineering", allocated to role "engineering/backend".

    {
      "allocation_info": { "role": "engineering/backend" },
      "id": <offer_id>,
      "framework_id": <framework_id>,
      "slave_id": <slave_id>,
      "hostname": <hostname>,
      "resources": [
        {
          "allocation_info": { "role": "engineering/backend" },
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 12 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "engineering",
              "principal": <principal>,
            }
          ]
        },
        {
          "allocation_info": { "role": "engineering/backend" },
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 6144 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "engineering",
              "principal": <principal>,
            }
          ]
        }
      ]
    }

Take note of the fact that role and reservation are not set, and that there is a new field called reservations which represents the reservation state. With RESERVATION_REFINEMENT enabled, the framework receives resources in this new format where solely the reservations field is used for the reservation state, rather than role/reservation pair from pre-RESERVATION_REFINEMENT.

We can reserve 8 CPUs and 4096 MB of RAM to "engineering/backend" by sending the following Offer::Operation message. Offer::Operation::Reserve has a resources field which we specify with the resources to be reserved. We must push a new ReservationInfo message onto the back of the reservations field. We must explicitly set the reservation's' role field to the offer's allocation role. The optional value of the principal field depends on whether or not the framework provided a principal when it registered with the master. If a principal was provided, then the resources' principal field must be equal to the framework's principal. If no principal was provided during registration, then the resources' principal field can take any value, or can be left unset. Note that the principal field determines the "reserver principal" when authorization is enabled, even if authentication is disabled.

    {
      "type": Offer::Operation::RESERVE,
      "reserve": {
        "resources": [
          {
            "allocation_info": { "role": "engineering/backend" },
            "name": "cpus",
            "type": "SCALAR",
            "scalar": { "value": 8 },
            "reservations": [
              {
                "type": "DYNAMIC",
                "role": "engineering",
                "principal": <principal>,
              },
              {
                "type": "DYNAMIC",
                "role": "engineering/backend",
                "principal": <framework_principal>,
              }
            ]
          },
          {
            "allocation_info": { "role": "engineering/backend" },
            "name": "mem",
            "type": "SCALAR",
            "scalar": { "value": 4096 },
            "reservations": [
              {
                "type": "DYNAMIC",
                "role": "engineering",
                "principal": <principal>,
              },
              {
                "type": "DYNAMIC",
                "role": "engineering/backend",
                "principal": <framework_principal>,
              }
            ]
          }
        ]
      }
    }

If the reservation is successful, a subsequent resource offer will contain the following reserved resources:

    {
      "allocation_info": { "role": "engineering/backend" },
      "id": <offer_id>,
      "framework_id": <framework_id>,
      "slave_id": <slave_id>,
      "hostname": <hostname>,
      "resources": [
        {
          "allocation_info": { "role": "engineering/backend" },
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 8 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "engineering",
              "principal": <principal>,
            },
            {
              "type": "DYNAMIC",
              "role": "engineering/backend",
              "principal": <framework_principal>,
            }
          ]
        },
        {
          "allocation_info": { "role": "engineering/backend" },
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 4096 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "engineering",
              "principal": <principal>,
            },
            {
              "type": "DYNAMIC",
              "role": "engineering/backend",
              "principal": <framework_principal>,
            }
          ]
        },
      ]
    }

Offer::Operation::Unreserve (with RESERVATION_REFINEMENT)

A framework can unreserve resources through the resource offer cycle. In Offer::Operation::Reserve, we reserved 8 CPUs and 4096 MB of RAM on a particular slave for one of our subscribed roles (i.e. "engineering/backend"), previously reserved for "engineering". When we unreserve these resources, they are returned to "engineering", by the last ReservationInfo added to the reservations field being popped. First, we receive a resource offer (copy/pasted from above):

    {
      "allocation_info": { "role": "engineering/backend" },
      "id": <offer_id>,
      "framework_id": <framework_id>,
      "slave_id": <slave_id>,
      "hostname": <hostname>,
      "resources": [
        {
          "allocation_info": { "role": "engineering/backend" },
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 8 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "engineering",
              "principal": <principal>,
            },
            {
              "type": "DYNAMIC",
              "role": "engineering/backend",
              "principal": <framework_principal>,
            }
          ]
        },
        {
          "allocation_info": { "role": "engineering/backend" },
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 4096 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "engineering",
              "principal": <principal>,
            },
            {
              "type": "DYNAMIC",
              "role": "engineering/backend",
              "principal": <framework_principal>,
            }
          ]
        },
      ]
    }

We can unreserve the 8 CPUs and 4096 MB of RAM by sending the following Offer::Operation message. Offer::Operation::Unreserve has a resources field which we can use to specify the resources to be unreserved.

    {
      "type": Offer::Operation::UNRESERVE,
      "unreserve": {
        "resources": [
          {
            "allocation_info": { "role": "engineering/backend" },
            "name": "cpus",
            "type": "SCALAR",
            "scalar": { "value": 8 },
            "reservations": [
              {
                "type": "DYNAMIC",
                "role": "engineering",
                "principal": <principal>,
              },
              {
                "type": "DYNAMIC",
                "role": "engineering/backend",
                "principal": <framework_principal>,
              }
            ]
          },
          {
            "allocation_info": { "role": "engineering/backend" },
            "name": "mem",
            "type": "SCALAR",
            "scalar": { "value": 4096 },
            "reservations": [
              {
                "type": "DYNAMIC",
                "role": "engineering",
                "principal": <principal>,
              },
              {
                "type": "DYNAMIC",
                "role": "engineering/backend",
                "principal": <framework_principal>,
              }
            ]
          },
        ]
      }
    }

The resources will now be reserved for "engineering" again, and may now be offered to "engineering" role itself, or other roles under "engineering".

Operator HTTP Endpoints

As described above, dynamic reservations can be made by a framework scheduler, typically in response to a resource offer. However, dynamic reservations can also be created and deleted by sending HTTP requests to the /reserve and /unreserve endpoints, respectively. This capability is intended for use by operators and administrative tools.

/reserve (since 0.25.0)

Suppose we want to reserve 8 CPUs and 4096 MB of RAM for the ads role on a slave with id=<slave_id> (note that it is up to the user to find the ID of the slave that hosts the desired resources; the request will fail if sufficient unreserved resources cannot be found on the slave). In this case, the principal that must be included in the reservation field of the reserved resources depends on the status of HTTP authentication on the master. If HTTP authentication is enabled, then the principal in the reservation should match the authenticated principal provided in the request's HTTP headers. If HTTP authentication is disabled, then the principal in the reservation can take any value, or can be left unset. Note that the principal field determines the "reserver principal" when authorization is enabled, even if HTTP authentication is disabled.

We send an HTTP POST request to the master's /reserve endpoint like so:

    $ curl -i \
      -u <operator_principal>:<password> \
      -d slaveId=<slave_id> \
      -d resources='[
        {
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 8 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "ads",
              "principal": <operator_principal>,
            }
          ]
        },
        {
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 4096 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "ads",
              "principal": <operator_principal>,
            }
          ]
        }
      ]' \
      -X POST http://<ip>:<port>/master/reserve

The user receives one of the following HTTP responses:

• 202 Accepted: Request accepted (see below).
• 400 BadRequest: Invalid arguments (e.g., missing parameters).
• 401 Unauthorized: Unauthenticated request.
• 403 Forbidden: Unauthorized request.
• 409 Conflict: Insufficient resources to satisfy the reserve operation.

This endpoint returns the 202 ACCEPTED HTTP status code, which indicates that the reserve operation has been validated successfully by the master. The request is then forwarded asynchronously to the Mesos slave where the reserved resources are located. That asynchronous message may not be delivered or reserving resources at the slave might fail, in which case no resources will be reserved. To determine if a reserve operation has succeeded, the user can examine the state of the appropriate Mesos slave (e.g., via the slave's /state HTTP endpoint).

/unreserve (since 0.25.0)

Suppose we want to unreserve the resources that we dynamically reserved above. We can send an HTTP POST request to the master's /unreserve endpoint like so:

    $ curl -i \
      -u <operator_principal>:<password> \
      -d slaveId=<slave_id> \
      -d resources='[
        {
          "name": "cpus",
          "type": "SCALAR",
          "scalar": { "value": 8 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "ads",
              "principal": <reserver_principal>,
            }
          ]
        },
        {
          "name": "mem",
          "type": "SCALAR",
          "scalar": { "value": 4096 },
          "reservations": [
            {
              "type": "DYNAMIC",
              "role": "ads",
              "principal": <reserver_principal>,
            }
          ]
        }
      ]' \
      -X POST http://<ip>:<port>/master/unreserve

Note that reserver_principal is the principal that was used to make the reservation, while operator_principal is the principal that is attempting to perform the unreserve operation---in some cases, these principals might be the same. The operator_principal must be authorized to unreserve reservations made by reserver_principal.

The user receives one of the following HTTP responses:

• 202 Accepted: Request accepted (see below).
• 400 BadRequest: Invalid arguments (e.g., missing parameters).
• 401 Unauthorized: Unauthenticated request.
• 403 Forbidden: Unauthorized request.
• 409 Conflict: Insufficient resources to satisfy the unreserve operation.

This endpoint returns the 202 ACCEPTED HTTP status code, which indicates that the unreserve operation has been validated successfully by the master. The request is then forwarded asynchronously to the Mesos slave where the reserved resources are located. That asynchronous message may not be delivered or unreserving resources at the slave might fail, in which case no resources will be unreserved. To determine if an unreserve operation has succeeded, the user can examine the state of the appropriate Mesos slave (e.g., via the slave's /state HTTP endpoint).

title: Apache Mesos - Shared Persistent Volumes layout: documentation

Shared Persistent Volumes

Overview

By default, persistent volumes provide exclusive access: once a task is launched using a persistent volume, no other tasks can use that volume, and the volume will not appear in any resource offers until the task that is using it has finished.

In some cases, it can be useful to share a volume between multiple tasks running on the same agent. For example, this could be used to efficiently share a large data set between multiple data analysis tasks.

Creating Shared Volumes

Shared persistent volumes are created using the same workflow as normal persistent volumes: by starting with a reserved resource and applying a CREATE operation, either via the framework scheduler API or the /create-volumes HTTP endpoint. To create a shared volume, set the shared field during volume creation.

For example, suppose a framework subscribed to the "engineering" role receives a resource offer containing 2048MB of dynamically reserved disk:

{
  "allocation_info": { "role": "engineering" },
  "id" : <offer_id>,
  "framework_id" : <framework_id>,
  "slave_id" : <slave_id>,
  "hostname" : <hostname>,
  "resources" : [
    {
      "allocation_info": { "role": "engineering" },
      "name" : "disk",
      "type" : "SCALAR",
      "scalar" : { "value" : 2048 },
      "role" : "engineering",
      "reservation" : {
        "principal" : <framework_principal>
      }
    }
  ]
}

The framework can create a shared persistent volume using this disk resource via the following offer operation:

{
  "type" : Offer::Operation::CREATE,
  "create": {
    "volumes" : [
      {
        "allocation_info": { "role": "engineering" },
        "name" : "disk",
        "type" : "SCALAR",
        "scalar" : { "value" : 2048 },
        "role" : "engineering",
        "reservation" : {
          "principal" : <framework_principal>
        },
        "disk": {
          "persistence": {
            "id" : <persistent_volume_id>
          },
          "volume" : {
            "container_path" : <container_path>,
            "mode" : <mode>
          }
        },
        "shared" : {
        }
      }
    ]
  }
}

Note that the shared field has been set (to an empty JSON object), which indicates that the CREATE operation will create a shared volume.

Using Shared Volumes

To be eligible to receive resource offers that contain shared volumes, a framework must enable the SHARED_RESOURCES capability in the FrameworkInfo it provides when it registers with the master. Frameworks that do not enable this capability will not be offered shared resources.

When a framework receives a resource offer, it can determine whether a volume is shared by checking if the shared field has been set. Unlike normal persistent volumes, a shared volume that is in use by a task will continue to be offered to the frameworks subscribed to the volume's role; this gives those frameworks the opportunity to launch additional tasks that can access the volume. A framework can also launch multiple tasks that access the volume using a single ACCEPT call.

Note that Mesos does not provide any isolation or concurrency control between the tasks that are sharing a volume. Framework developers should ensure that tasks that access the same volume do not conflict with one another. This can be done via careful application-level concurrency control, or by ensuring that the tasks access the volume in a read-only manner. Mesos provides support for read-only access to volumes: as described in the persistent volume documentation, tasks that are launched on a volume can specify a mode of "RO" to use the volume in read-only mode.

Destroying Shared Volumes

A persistent volume, whether shared or not, can only be destroyed if no running or pending tasks have been launched using the volume. For non-shared volumes, it is usually easy to determine when it is safe to delete a volume. For shared volumes, the framework(s) that have launched tasks using the volume typically need to coordinate to ensure (e.g., via reference counting) that a volume is no longer being used before it is destroyed.

Resource Allocation

TODO: how do shared volumes influence resource allocation?

References

• MESOS-3421 contains additional information about the implementation of this feature.

• Talk at MesosCon Europe 2016 on August 31, 2016 entitled "Practical Persistent Volumes".

title: Apache Mesos - Oversubscription layout: documentation

Oversubscription

High-priority user-facing services are typically provisioned on large clusters for peak load and unexpected load spikes. Hence, for most of time, the provisioned resources remain underutilized. Oversubscription takes advantage of temporarily unused resources to execute best-effort tasks such as background analytics, video/image processing, chip simulations, and other low priority jobs.

How does it work?

Oversubscription was introduced in Mesos 0.23.0 and adds two new agent components: a Resource Estimator and a Quality of Service (QoS) Controller, alongside extending the existing resource allocator, resource monitor, and Mesos agent. The new components and their interactions are illustrated below.

Resource estimation

• (1) The first step is to identify the amount of oversubscribed resources. The resource estimator taps into the resource monitor and periodically gets usage statistics via ResourceStatistic messages. The resource estimator applies logic based on the collected resource statistics to determine the amount of oversubscribed resources. This can be a series of control algorithms based on measured resource usage slack (allocated but unused resources) and allocation slack.

• (2) The agent keeps polling estimates from the resource estimator and tracks the latest estimate.

• (3) The agent will send the total amount of oversubscribed resources to the master when the latest estimate is different from the previous estimate.

Resource tracking & scheduling algorithm

• (4) The allocator keeps track of the oversubscribed resources separately from regular resources and annotate those resources as revocable. It is up to the resource estimator to determine which types of resources can be oversubscribed. It is recommended only to oversubscribe compressible resources such as cpu shares, bandwidth, etc.

Frameworks

• (5) Frameworks can choose to launch tasks on revocable resources by using the regular launchTasks() API. To safe-guard frameworks that are not designed to deal with preemption, only frameworks registering with the REVOCABLE_RESOURCES capability set in its framework info will receive offers with revocable resources. Further more, revocable resources cannot be dynamically reserved and persistent volumes should not be created on revocable disk resources.

Task launch

• The revocable task is launched as usual when the runTask request is received on the agent. The resources will still be marked as revocable and isolators can take appropriate actions, if certain resources need to be setup differently for revocable and regular tasks.

NOTE: If any resource used by a task or executor is revocable, the whole container is treated as a revocable container and can therefore be killed or throttled by the QoS Controller.

Interference detection

• (6) When the revocable task is running, it is important to constantly monitor the original task running on those resources and guarantee performance based on an SLA. In order to react to detected interference, the QoS controller needs to be able to kill or throttle running revocable tasks.

Enabling frameworks to use oversubscribed resources

Frameworks planning to use oversubscribed resources need to register with the REVOCABLE_RESOURCES capability set:

FrameworkInfo framework;
framework.set_name("Revocable framework");

framework.add_capabilities()->set_type(
    FrameworkInfo::Capability::REVOCABLE_RESOURCES);

From that point on, the framework will start to receive revocable resources in offers.

NOTE: That there is no guarantee that the Mesos cluster has oversubscription enabled. If not, no revocable resources will be offered. See below for instructions how to configure Mesos for oversubscription.

Launching tasks using revocable resources

Launching tasks using revocable resources is done through the existing launchTasks API. Revocable resources will have the revocable field set. See below for an example offer with regular and revocable resources.

{
  "id": "20150618-112946-201330860-5050-2210-0000",
  "framework_id": "20141119-101031-201330860-5050-3757-0000",
  "agent_id": "20150618-112946-201330860-5050-2210-S1",
  "hostname": "foobar",
  "resources": [
    {
      "name": "cpus",
      "type": "SCALAR",
      "scalar": {
        "value": 2.0
      },
      "role": "*"
    }, {
      "name": "mem",
      "type": "SCALAR",
      "scalar": {
        "value": 512.0
      },
      "role": "*"
    },
    {
      "name": "cpus",
      "type": "SCALAR",
      "scalar": {
        "value": 0.45
      },
      "role": "*",
      "revocable": {}
    }
  ]
}

Writing a custom resource estimator

The resource estimator estimates and predicts the total resources used on the agent and informs the master about resources that can be oversubscribed. By default, Mesos comes with a noop and a fixed resource estimator. The noop estimator only provides an empty estimate to the agent and stalls, effectively disabling oversubscription. The fixed estimator doesn't use the actual measured slack, but oversubscribes the node with fixed resource amount (defined via a command line flag).

The interface is defined below:

class ResourceEstimator
{
public:
  // Initializes this resource estimator. This method needs to be
  // called before any other member method is called. It registers
  // a callback in the resource estimator. The callback allows the
  // resource estimator to fetch the current resource usage for each
  // executor on agent.
  virtual Try<Nothing> initialize(
      const lambda::function<process::Future<ResourceUsage>()>& usage) = 0;

  // Returns the current estimation about the *maximum* amount of
  // resources that can be oversubscribed on the agent. A new
  // estimation will invalidate all the previously returned
  // estimations. The agent will be calling this method periodically
  // to forward it to the master. As a result, the estimator should
  // respond with an estimate every time this method is called.
  virtual process::Future<Resources> oversubscribable() = 0;
};

Writing a custom QoS controller

The interface for implementing custom QoS Controllers is defined below:

class QoSController
{
public:
  // Initializes this QoS Controller. This method needs to be
  // called before any other member method is called. It registers
  // a callback in the QoS Controller. The callback allows the
  // QoS Controller to fetch the current resource usage for each
  // executor on agent.
  virtual Try<Nothing> initialize(
      const lambda::function<process::Future<ResourceUsage>()>& usage) = 0;

  // A QoS Controller informs the agent about corrections to carry
  // out, but returning futures to QoSCorrection objects. For more
  // information, please refer to mesos.proto.
  virtual process::Future<std::list<QoSCorrection>> corrections() = 0;
};

NOTE The QoS Controller must not block corrections(). Back the QoS Controller with its own libprocess actor instead.

The QoS Controller informs the agent that particular corrective actions need to be made. Each corrective action contains information about executor or task and the type of action to perform.

Mesos comes with a noop and a load qos controller. The noop controller does not provide any corrections, thus does not assure any quality of service for regular tasks. The load controller is ensuring the total system load doesn't exceed a configurable thresholds and as a result try to avoid the cpu congestion on the node. If the load is above the thresholds controller evicts all the revocable executors. These thresholds are configurable via two module parameters load_threshold_5min and load_threshold_15min. They represent standard unix load averages in the system. 1 minute system load is ignored, since for oversubscription use case it can be a misleading signal.

message QoSCorrection {
  enum Type {
    KILL = 1; // Terminate an executor.
  }

  message Kill {
    optional FrameworkID framework_id = 1;
    optional ExecutorID executor_id = 2;
  }

  required Type type = 1;
  optional Kill kill = 2;
}

Configuring oversubscription

Five new flags has been added to the agent:

The fixed resource estimator is enabled as follows:

--resource_estimator="org_apache_mesos_FixedResourceEstimator"

--modules='{
  "libraries": {
    "file": "/usr/local/lib64/libfixed_resource_estimator.so",
    "modules": {
      "name": "org_apache_mesos_FixedResourceEstimator",
      "parameters": {
        "key": "resources",
        "value": "cpus:14"
      }
    }
  }
}'

In the example above, a fixed amount of 14 cpus will be offered as revocable resources.

The load qos controller is enabled as follows:

--qos_controller="org_apache_mesos_LoadQoSController"

--qos_correction_interval_min="20secs"

--modules='{
  "libraries": {
    "file": "/usr/local/lib64/libload_qos_controller.so",
    "modules": {
      "name": "org_apache_mesos_LoadQoSController",
      "parameters": [
        {
          "key": "load_threshold_5min",
          "value": "6"
        },
        {
	  "key": "load_threshold_15min",
	  "value": "4"
        }
      ]
    }
  }
}'

In the example above, when standard unix system load average for 5 minutes will be above 6, or for 15 minutes will be above 4 then agent will evict all the revocable executors. LoadQoSController will be effectively run every 20 seconds.

To install a custom resource estimator and QoS controller, please refer to the modules documentation.

title: Apache Mesos - Authentication layout: documentation

Authentication

Authentication permits only trusted entities to interact with a Mesos cluster. Authentication can be used by Mesos in three ways:

1. To require that frameworks be authenticated in order to register with the master.
2. To require that agents be authenticated in order to register with the master.
3. To require that operators be authenticated to use many HTTP endpoints.

Authentication is disabled by default. When authentication is enabled, operators can configure Mesos to either use the default authentication module or to use a custom authentication module.

The default Mesos authentication module uses the Cyrus SASL library. SASL is a flexible framework that allows two endpoints to authenticate with each other using a variety of methods. By default, Mesos uses CRAM-MD5 authentication.

Credentials, Principals, and Secrets

When using the default CRAM-MD5 authentication method, an entity that wants to authenticate with Mesos must provide a credential, which consists of a principal and a secret. The principal is the identity that the entity would like to use; the secret is an arbitrary string that is used to verify that identity. Principals are similar to user names, while secrets are similar to passwords.

Principals are used primarily for authentication and authorization; note that a principal is different from a framework's user, which is the operating system account used by the agent to run executors, and the framework's roles, which are used to determine which resources a framework can use.

Configuration

Authentication is configured by specifying command-line flags when starting the Mesos master and agent processes. For more information, refer to the configuration documentation.

Master

• --[no-]authenticate - If true, only authenticated frameworks are allowed to register. If false (the default), unauthenticated frameworks are also allowed to register.

• --[no-]authenticate_http_readonly - If true, authentication is required to make HTTP requests to the read-only HTTP endpoints that support authentication. If false (the default), these endpoints can be used without authentication. Read-only endpoints are those which cannot be used to modify the state of the cluster.

• --[no-]authenticate_http_readwrite - If true, authentication is required to make HTTP requests to the read-write HTTP endpoints that support authentication. If false (the default), these endpoints can be used without authentication. Read-write endpoints are those which can be used to modify the state of the cluster.

• --[no-]authenticate_agents - If true, only authenticated agents are allowed to register. If false (the default), unauthenticated agents are also allowed to register.

• --authentication_v0_timeout - The timeout within which an authentication is expected to complete against a v0 framework or agent. This does not apply to the v0 or v1 HTTP APIs.(default: 15secs)

• --authenticators - Specifies which authenticator module to use. The default is crammd5, but additional modules can be added using the --modules option.

• --http_authenticators - Specifies which HTTP authenticator module to use. The default is basic (basic HTTP authentication), but additional modules can be added using the --modules option.

• --credentials - The path to a text file which contains a list of accepted credentials. This may be optional depending on the authenticator being used.