Airflow Mesos Provider
The Airflow Mesos Provider integrates Apache Airflow with Apache Mesos.
It provides two execution models:
- MesosExecutor: distributes regular Airflow task workloads through a Mesos framework.
- MesosOperator: starts a single DAG task as a Mesos container and waits for it to finish.
Quick start
- Install the provider and configure Airflow: Installation
- Understand the responsibilities and data flow: Architecture
- Use the MesosOperator for individual container tasks.
- Read the DAG examples for complete examples.
- Verify the setup with Testing and development.
Requirements
- Apache Airflow 3.x recommended; the provider declares Airflow
>=2.0. - Apache Mesos 1.6 or newer.
- Python 3.x.
- For Docker containers: a Mesos agent with the Docker containerizer enabled and access to the required image.
SSL and Mesos authentication are optional, but recommended for production clusters.