Deferrable and Async Operators in Airflow

Architecture Diagram

Formal Definitions

Detailed Explanation

The Problem with Blocking Operators

When a traditional operator waits for an external event (like an API response or file availability), it holds a worker slot the entire time. This wastes resources because the worker is doing nothing but polling.

Key Insight: A blocking sensor polling every 30 seconds for 1 hour uses only 0.1% of its time productively — the other 99.9% is wasted polling.

How Deferrable Operators Solve This

Why Deferrable Operators?

Traditional blocking operators hold a worker slot while waiting for external conditions. For long-running operations like API callbacks, database queries, or cloud job completions, this wastes valuable executor resources. Deferrable operators release the worker slot immediately and delegate the wait to the lightweight triggerer process.

For a blocking sensor polling every 30 seconds for 1 hour: (nearly all time is wasted polling). A deferrable equivalent achieves since the worker slot is released immediately.

Creating a Deferrable Operator

HTTP Polling Trigger

Triggerer Configuration

Key Concepts Table

Comparison: Blocking vs Deferrable

Code Examples

Custom Trigger with Multiple Conditions

Deferrable Sensor Pattern

Performance Metrics

Best Practices

Implementation Guidelines

1. Use for long waits: Deferrable operators shine when tasks wait >5 minutes for external events.
2. Implement serialize(): Always implement serialize() on triggers for persistence across triggerer restarts.
3. Handle timeouts: Set execution_timeout on defer calls to prevent infinite waits.
4. Use async libraries: Leverage aiohttp, httpx, or aiobotocore for non-blocking I/O in triggers.

Operational Guidelines

1. Monitor triggerer capacity: Track triggers.running and triggers.queued metrics.
2. Fail gracefully: Handle trigger errors by yielding TriggerEvent({'status': 'error'}).
3. Avoid blocking in triggers: Never use time.sleep() or synchronous I/O in trigger run() methods.
4. Test triggers independently: Mock the async loop and verify trigger serialization.

When to Use Deferrable Operators

See Also

• Sensors and Operators — Blocking sensors and poke modes
• Scheduling and Triggers — Timetables and scheduling patterns
• Operators and Hooks — Operator lifecycle and hook architecture
• Executors Comparison — Executor selection and resource management