Dynamic DAG Generation in Airflow

Architecture Diagram

Formal Definitions

Detailed Explanation

Why Dynamic DAGs?

Static DAG definitions work well for small deployments. As the number of pipelines grows (hundreds or thousands), manual DAG definitions become unsustainable. Dynamic generation lets you define pipeline logic once and instantiate it across multiple datasets, teams, or environments from configuration.

Key Insight: Dynamic DAGs reduce code duplication and make it easier to maintain consistent pipeline patterns across your organization.

Configuration Sources

DAG Generation Lifecycle

1. Scheduler scans dags/ folder for Python files
2. Python file is parsed — all top-level code executes
3. Factory function runs — reads configuration
4. Config items are looped — creates DAG objects
5. DAGs are registered — added to globals() for scheduler discovery

Pattern 1: YAML-Driven Generation

Pipeline Configuration YAML

# /opt/airflow/configs/pipelines.yaml
- name: orders_daily
  owner: orders-team
  start_date: "2024-01-01"
  schedule: "0 2 * * *"
  source_conn: source_postgres
  target_conn: warehouse_postgres
  target_table: fct_orders
  extract_sql: |
    SELECT * FROM raw_orders
    WHERE date = '{{ ds }}'
  transform_sql: |
    INSERT INTO stg_orders
    SELECT order_id, customer_id, amount * 1.1 as adjusted_amount
    FROM raw_orders WHERE date = '{{ ds }}'
  load_sql: |
    INSERT INTO fct_orders
    SELECT * FROM stg_orders WHERE date = '{{ ds }}'
  quality_rules:
    not_null: "SELECT COUNT(*) FROM fct_orders WHERE order_id IS NULL"
    positive_amount: "SELECT COUNT(*) FROM fct_orders WHERE amount <= 0"
  tags: ["orders", "daily"]

- name: customers_weekly
  owner: customer-team
  start_date: "2024-01-01"
  schedule: "0 6 * * 1"
  source_conn: source_mysql
  target_conn: warehouse_postgres
  target_table: dim_customers
  extract_sql: "SELECT * FROM customers WHERE updated_at >= '{{ ds }}'"
  transform_sql: "INSERT INTO dim_customers SELECT * FROM stg_customers"
  load_sql: "SELECT 1"
  quality_rules: {}
  tags: ["customers", "weekly"]

Pattern 2: Database-Driven Generation

Pattern 3: Loop-Based Generation

Key Concepts Table

Performance Metrics

Best Practices

Configuration Management

1. Idempotent generation: Always produce equivalent DAGs from the same config. Avoid time-dependent values in DAG definitions.
2. Cache config reads: When reading from databases or APIs, cache results during parse to avoid repeated calls.
3. Validate configs: Add schema validation for YAML/database configs before DAG generation.
4. Version configs: Track configuration changes alongside code changes for reproducibility.

Performance Optimization

1. Set max_active_runs: Prevent resource exhaustion when many generated DAGs trigger simultaneously.
2. Use globals() registration: Generated DAGs must be added to globals() for the scheduler to discover them.
3. Monitor parse time: Dynamic generation increases parse overhead. Keep config reads fast and cacheable.

Operational Guidelines

1. Tag generated DAGs: Include source config identifiers in tags for monitoring and filtering.

Common Mistakes

See Also

• DAG Design Patterns — DAG composition and dependency patterns
• Complex Multi-DAG Orchestration — Cross-DAG coordination patterns
• Scheduling and Triggers — Timetables and scheduling patterns
• Airflow Architecture — Core architecture and parse lifecycle