Performance Tuning and Optimization
Architecture Diagram
Formal Definitions
Detailed Explanation
Scheduler Optimization
Key Scheduler Settings:
| Parameter | Default | Recommended | Description |
|---|---|---|---|
min_file_process_interval | 30 | 30-60 | Seconds between DAG file scans |
dag_dir_list_interval | 300 | 300-600 | Seconds between directory listings |
parsing_processes | 2 | 2-4 | Parallel DAG parsing processes |
scheduler_heartbeat_sec | 5 | 5 | Scheduler heartbeat interval |
parallelism | 32 | 32-128 | Max concurrent tasks |
max_active_tasks_per_dag | 16 | 16-64 | Max tasks per DAG |
max_active_runs_per_dag | 16 | 16-32 | Max DAG runs per DAG |
[scheduler]
min_file_process_interval = 30
parsing_processes = 2
parallelism = 32
max_active_tasks_per_dag = 16
store_serialized_dags = True
Database Optimization
Essential Indexes:
| Index | Table | Purpose |
|---|---|---|
idx_task_instance_dag_run | task_instance | Speeds up DAG run queries |
idx_task_instance_state | task_instance | Fast state filtering |
idx_dag_run_state | dag_run | Fast DAG run state queries |
Connection Pool Settings:
| Parameter | Recommended | Description |
|---|---|---|
pool_size | 20 | Base connections |
max_overflow | 30 | Extra connections for peaks |
pool_timeout | 30 | Wait time for connection |
pool_recycle | 1800 | Recycle after 30 min |
pool_pre_ping | True | Verify connections |
[database]
sql_alchemy_pool_size = 20
sql_alchemy_max_overflow = 30
sql_alchemy_pool_recycle = 1800
sql_alchemy_pool_pre_ping = True
Worker Optimization
Celery Worker Settings:
| Parameter | Description | Recommendation |
|---|---|---|
worker_concurrency | Tasks per worker | 4-16 (CPU-bound: 4-8, I/O-bound: 16-32) |
worker_prefetch_multiplier | Tasks prefetched | 1 (fair scheduling) |
worker_max_tasks_per_child | Recycle worker after N tasks | 1000-2000 |
task_acks_late | Ack after execution | True (fault tolerance) |
[celery]
worker_concurrency = 16
worker_prefetch_multiplier = 1
worker_max_tasks_per_child = 1000
task_acks_late = True
Tip: For CPU-bound tasks, use lower concurrency (4-8). For I/O-bound, use higher (16-32).
return pool_config
Architecture Diagram
### Worker Optimization
```python
# worker_optimization.py
import psutil
import os
def get_worker_recommendations():
"""Get resource recommendations based on system specs."""
cpu_count = psutil.cpu_count()
memory = psutil.virtual_memory()
# Celery worker configuration
worker_config = {
# Concurrency = CPU cores (for CPU-bound tasks)
# Concurrency = 2 * CPU cores (for I/O-bound tasks)
'concurrency': min(cpu_count, 16),
# Prefetch multiplier - how many tasks to prefetch
'prefetch_multiplier': 1,
# Maximum tasks per child before worker restart
'max_tasks_per_child': 200,
# Worker memory limit
'max_memory_per_child': int(memory.total * 0.8 / cpu_count),
# Task time limit (seconds)
'task_time_limit': 3600,
# Soft time limit (seconds) - raises SoftTimeLimitExceeded
'task_soft_time_limit': 3000,
}
return worker_config
def monitor_worker_health():
"""Monitor worker health metrics."""
import psutil
metrics = {
'cpu_percent': psutil.cpu_percent(interval=1),
'memory_percent': psutil.virtual_memory().percent,
'disk_usage': psutil.disk_usage('/').percent,
'open_files': len(psutil.Process().open_files()),
'connections': len(psutil.Process().connections()),
}
# Alert thresholds
alerts = []
if metrics['cpu_percent'] > 90:
alerts.append(f"High CPU: {metrics['cpu_percent']}%")
if metrics['memory_percent'] > 85:
alerts.append(f"High Memory: {metrics['memory_percent']}%")
if metrics['disk_usage'] > 90:
alerts.append(f"High Disk: {metrics['disk_usage']}%")
return {
'metrics': metrics,
'alerts': alerts,
'healthy': len(alerts) == 0,
}
Key Concepts Table
| Optimization Area | Metric | Target | Impact |
|---|---|---|---|
| DAG Parsing | Parse time | < 1s per DAG | High |
| Task Latency | Queue to start | < 5s | High |
| DB Query Time | Average query | < 100ms | High |
| Worker Memory | Per-worker | < 4GB | Medium |
| XCom Size | Per operation | < 48KB | Medium |
| Log Storage | Daily volume | < 10GB/day | Low |
| Scheduler Heartbeat | Interval | 5s | Low |
Code Examples
Performance Monitoring Dashboard
DAG Optimization Patterns
Resource-Aware Task Scheduling
Performance Metrics
Optimization Impact
| Optimization | Before | After | Improvement |
|---|---|---|---|
| DAG Serialization | 10s parse | 2s parse | 80% faster |
| DB Indexing | 500ms query | 50ms query | 90% faster |
| Connection Pooling | 100ms connect | 10ms connect | 90% faster |
| Worker Concurrency | 4 tasks | 16 tasks | 4x throughput |
| XCom Backend | 500ms push | 50ms push | 90% faster |
Resource Utilization
| Resource | Recommended | Warning | Critical |
|---|---|---|---|
| CPU | < 70% | 70-85% | > 85% |
| Memory | < 70% | 70-85% | > 85% |
| Disk I/O | < 70% | 70-85% | > 85% |
| Network | < 50% | 50-80% | > 80% |
| DB Connections | < 70% | 70-85% | > 85% |
See Also
- Airflow Architecture â Core architecture components
- Executors Comparison â Choosing the right executor
- Kubernetes Executor â Dynamic scaling with K8s
- Monitoring and Alerting â Performance monitoring