Complex Multi-DAG Orchestration
Architecture Diagram
Formal Definitions
Detailed Explanation
Multi-DAG Orchestration Patterns
| Pattern | Mechanism | Wait Type | Complexity | Best For |
|---|---|---|---|---|
| TriggerDagRunOperator | Creates child dag_run | Blocking or non-blocking | Low | Simple parent-child |
| ExternalTaskSensor | Polls child task state | Reschedule mode recommended | Medium | Implicit dependencies |
| Callback-based | DAG failure/success hooks | Asynchronous notification | Medium | Alerting and logging |
| Dynamic DAGs | Code-generated dependencies | Variable | High | Multi-team coordination |
| Fan-Out/Fan-In | Parallel triggers + consolidated wait | Parallel with barrier | Medium | Multi-region/multi-table |
When to Use Each Pattern
- TriggerDagRunOperator: When you need to trigger a child DAG and optionally wait for completion
- ExternalTaskSensor: When you need to wait for a specific task in another DAG to complete
- Callbacks: When you need to be notified of child DAG success/failure without polling
- Fan-Out/Fan-In: When you need to process multiple datasets in parallel and consolidate results
Pattern 1: TriggerDagRunOperator
The simplest way to invoke a child DAG from a parent. The parent triggers the child and can optionally wait for completion.
Pattern 2: ExternalTaskSensor
Wait for a specific task in another DAG to reach a terminal state. This creates implicit inter-DAG dependencies.
Pattern 3: Cross-DAG XCom with Dynamic DAGs
Pattern 4: Callback-Driven Coordination
Pattern 5: Fan-Out / Fan-In with Multiple DAGs
Key Concepts Table
| Pattern | Mechanism | Wait Type | Complexity | Best For |
|---|---|---|---|---|
| TriggerDagRunOperator | Creates child dag_run | Blocking or non-blocking | Low | Simple parent-child |
| ExternalTaskSensor | Polls child task state | Reschedule mode recommended | Medium | Implicit dependencies |
| Callback-based | DAG failure/success hooks | Asynchronous notification | Medium | Alerting and logging |
| Dynamic DAGs | Code-generated dependencies | Variable | High | Multi-team coordination |
| Fan-Out/Fan-In | Parallel triggers + consolidated wait | Parallel with barrier | Medium | Multi-region/multi-table |
Performance Metrics
| Metric | TriggerDagRun | ExternalTaskSensor | Callbacks |
|---|---|---|---|
| Latency to trigger | <1s | N/A (polling) | Immediate |
| Worker slot during wait | Released (if wait=False) | Held (unless reschedule) | N/A |
| Scalability | High | Medium (sensor overhead) | High |
| Error handling | Task failure | Sensor timeout | Event-driven |
Best Practices
Pattern Selection
- Use
mode='reschedule'on ExternalTaskSensor to avoid holding worker slots during waits. - Set
execution_deltaorexecution_date_fnto match the child DAG's execution schedule. - Implement
wait_for_completion=Trueon TriggerDagRunOperator when downstream tasks depend on child completion.
Communication and Monitoring
- Use callbacks for alerting and logging â they fire independently of task state transitions.
- Avoid deep inter-DAG chains â they create fragile, hard-to-debug dependencies.
- Use XCom sparingly across DAGs â prefer external state stores for large data.
Operational Guidelines
- Tag orchestration DAGs distinctly from child DAGs for monitoring clarity.
- Set
execution_timeouton sensors and triggers to prevent indefinite hangs.
Common Pitfalls
| Pitfall | Problem | Solution |
|---|---|---|
| Deep dependency chains | Hard to debug, fragile | Keep to 2-3 levels max |
| Missing timeouts | Sensors hang indefinitely | Always set execution_timeout |
| Blocking sensors | Wastes worker slots | Use mode='reschedule' |
| XCom for large data | Database bloat | Use external storage (S3, GCS) |
See Also
- DAG Design Patterns â DAG composition and dependency patterns
- Scheduling and Triggers â Timetables and scheduling patterns
- Branching Logic â BranchPythonOperator and conditional workflows
- XCom Communications â Task communication and data passing