Variable Management and Templates
Architecture Diagram
Formal Definitions
Detailed Explanation
Variable Basics
Variables provide a simple way to store configuration values across DAGs and tasks.
Operations:
| Operation | Method | Example |
|---|---|---|
| Get | Variable.get(key, default_var) | Variable.get("api_key", "default") |
| Get JSON | Variable.get(key, deserialize_json=True) | Returns Python dict |
| Set | Variable.set(key, value) | Variable.set("config", {"key": "val"}) |
| CLI Get | airflow variables get my_variable | Command line |
| CLI Set | airflow variables set my_variable "value" | Command line |
Jinja Template Variables
Airflow provides built-in template variables for task parameters.
Common Template Variables:
| Variable | Description |
|---|---|
{{ ds }} | Execution date (YYYY-MM-DD) |
{{ ds_nodash }} | Execution date (YYYYMMDD) |
{{ ts }} | Execution timestamp |
{{ prev_execution_date }} | Previous execution date |
{{ params }} | DAG run parameters |
{{ ds_nodash }} | Date without dashes |
SELECT * FROM events
WHERE event_date = '{{ ds }}'
AND hour = {{ params.hour }}
Variable Best Practices
| Practice | Description |
|---|---|
| Avoid in Loop | Don't call Variable.get() in DAG top-level â causes DB query per parse |
| Use JSON | Store complex configs as JSON variables |
| Secrets Backend | Use Vault/AWS Secrets Manager for sensitive values |
| Default Values | Always provide default_var for optional variables |
Variable Templates
Key Concepts Table
| Variable Type | Storage | Access Method | Use Case |
|---|---|---|---|
| String | Metadata DB | Variable.get("key") | Simple config |
| JSON | Metadata DB | Variable.get("key", deserialize_json=True) | Complex config |
| Environment | OS Environment | Variable.get("key") with env prefix | Secrets, Docker |
| Secrets Backend | External vault | Variable.get("key") with backend | Production secrets |
| DAG Parameters | DAG run | {{ params.key }} | Runtime config |
Code Examples
Advanced Templating Patterns
Variable Caching Strategy
Variable-Based DAG Configuration
Performance Metrics
Variable Access Patterns
| Access Pattern | Latency | Use Case |
|---|---|---|
| Direct Variable.get() | ~5ms | One-time access |
| Cached access | ~1Ξs | Repeated access |
| Template variable | ~0ms (pre-rendered) | DAG parameter |
| Secrets Backend | ~50ms | Secure values |
Variable vs Parameter Comparison
| Feature | Variable | DAG Parameter | Template Variable |
|---|---|---|---|
| Storage | Metadata DB | DAG Run | Pre-rendered |
| Scope | Global | Per-run | Per-task |
| Mutability | Mutable | Immutable | Immutable |
| Access Pattern | Variable.get() | params.key | {{ var.value.key }} |
| Performance | ~5ms | ~0ms | ~0ms |
| Use Case | Shared config | Runtime config | Task config |
See Also
- XCom Communications â Task-to-task data passing
- Connection Management â External system connections
- Security Best Practices â Secrets management
- DAG Design Patterns â DAG configuration patterns