AirflowのDAGやtaskの同時実行数

parallelism

Airflow 全体での task instance の並列実行数

dag_concurrency

task instance ごとの並行実行数

max_active_runs_per_dag

DAG ごとの DAG run の同時実行数の最大値

code:airflow.cfg

# The amount of parallelism as a setting to the executor. This defines

# the max number of task instances that should run simultaneously on this airflow installation

parallelism = 32

# The number of task instances allowed to run concurrently by the scheduler

dag_concurrency = 16

# The maximum number of active DAG runs per DAG

max_active_runs_per_dag = 16

DAGファイルでも記述可能

code:python

dag = DAG(

...

concurrency=5, # このDAGのtaskは5つまで並行実行できる

max_active_runs=2, # このDAGは2つまで同時起動できる

)