Airflow Mesos Provider

Introduction

There are two parts of the "Airflow Mesos Provider".

  • Mesos Scheduler: These provide the possibility to schedule Airflow tasks over the Mesos infrastructure
  • Mesos Operator: These will run the specific DAG Task direct as Mesos task.

Requirements

  • Apache Mesos min 1.6.0
  • Mesos with SSL and Authentication is optional
  • Airflow 2.x
  • Python 3.x

How to install the AV Mesos Airflow Provider

To install the Mesos Airflow Provider, you have to to the following steps:

  1. Install the provider
pip install avmesos_airflow_provider
  1. Configure Airflow to use the new executor
vim airflow.cfg

[core]
executor = avmesos_airflow_provider.executors.mesos_executor.MesosExecutor

[mesos]
mesos_ssl = False
master = leader.mesos:5050
framework_name = Airflow
checkpoint = True
failover_timeout = 604800
command_shell = True
task_cpu = 1
task_memory = 20000
authenticate = True
default_principal = <MESOS USER>
default_secret = <MESOS PASSWORD>
docker_image_slave = <AIRFLOW DOCKER IMAGE>
docker_volume_driver = local
docker_volume_dag_name = airflowdags
docker_volume_dag_container_path = /home/airflow/airflow/dags/
docker_sock = /var/run/docker.sock
docker_volume_logs_name = airflowlogs
docker_volume_logs_container_path = /home/airflow/airflow/logs/
docker_environment = '{ "name":"<KEY>", "value":"<VALUE>" }, { ... },' // << do not forget the comma at the end.

DAG example with mesos executor

from airflow import DAG
from datetime import datetime, timedelta
from airflow.operators.dummy_operator import DummyOperator
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.operators.python import PythonOperator

default_args = {
        'owner'                 : 'airflow',
        'description'           : 'Use of the DockerOperator',
        'depend_on_past'        : True,
}

with DAG('docker_dag2', default_args=default_args, schedule_interval="*/10 * * * * ", catchup=True, start_date=datetime.now()) as dag:
        t2 = DockerOperator(
                task_id='docker_command',
                image='centos:latest',
                api_version='auto',
                auto_remove=False,
                command="/bin/sleep 600",
                docker_url='unix:///var/run/docker.sock'
                executor_config={
                        "MesosExecutor": {
                                "cpus": 2.0,
                                "mem_limit": 2048
                        }
                }         
        )

        t2