Error Handling and Retry Strategies
Architecture Diagram
Formal Definitions
Detailed Explanation
Retry Configuration
Airflow provides built-in retry mechanisms for handling transient failures.
Retry Parameters:
| Parameter | Description | Example |
|---|---|---|
retries | Number of retry attempts | 3 |
retry_delay | Time between retries | timedelta(minutes=5) |
retry_exponential_backoff | Double delay each retry | True |
max_retry_delay | Maximum retry delay cap | timedelta(minutes=30) |
execution_timeout | Max task execution time | timedelta(hours=1) |
Callback Functions
Callbacks are invoked on specific task state transitions.
Callback Types:
| Callback | Trigger | Use Case |
|---|---|---|
on_failure_callback | Task fails | Send alerts, log errors |
on_success_callback | Task succeeds | Notify, update external systems |
on_retry_callback | Task retries | Log retry attempts |
on_execute_callback | Task starts | Initialize resources |
SLA Configuration
SLAs define expected completion times for tasks.
SLA Parameters:
| Level | Setting | Effect |
|---|---|---|
| DAG-level | sla_miss_callback | Invoked when any task misses SLA |
| Task-level | sla=timedelta(hours=2) | Per-task SLA |
SLA Violation Condition: T_completion > T_sla + T_execution_date