Connection Management and Hooks
Architecture Diagram
Formal Definitions
Detailed Explanation
Creating Connections
Connections can be created through the Airflow UI, CLI, REST API, or environment variables.
Creation Methods:
| Method | Command/Location |
|---|---|
| CLI | airflow connections add 'my_conn' --conn-type ... |
| Environment Variable | AIRFLOW_CONN_MY_CONN='...' |
| UI | Admin â Connections â Add |
| Programmatic | Connection(conn_id=..., conn_type=...) |
Using Hooks
Hooks provide a consistent interface for external system interaction.
Tip: Use connection pools to reduce overhead by 60-80%.
Key Concepts Table
| Connection Type | Conn Type | Hook Class | Key Parameters |
|---|---|---|---|
| PostgreSQL | postgres | PostgresHook | host, login, password, schema, port |
| MySQL | mysql | MySqlHook | host, login, password, schema, port |
| S3 | aws | S3Hook | login (key), password (secret), extra (region) |
| GCS | google_cloud_platform | GCSHook | extra (keyfile, project) |
| HTTP | http | HttpHook | host, schema, extra (headers) |
Key Concepts Table
| Connection Type | Conn Type | Hook Class | Key Parameters |
|---|---|---|---|
| PostgreSQL | postgres | PostgresHook | host, login, password, schema, port |
| MySQL | mysql | MySqlHook | host, login, password, schema, port |
| S3 | aws | S3Hook | login (key), password (secret), extra (region) |
| GCS | google_cloud_platform | GCSHook | extra (keyfile, project) |
| HTTP | http | HttpHook | host, schema, extra (headers) |
| Spark | spark | SparkHook | host, extra (master, deploy-mode) |
| Redis | redis | RedisHook | host, login, password, port, db |
Code Examples
Custom Hook Implementation
Connection Pool Configuration
Secrets Backend Integration
Performance Metrics
Connection Types Comparison
| Aspect | Metadata DB | Environment Variables | Secrets Manager |
|---|---|---|---|
| Security | Medium | High | Very High |
| Performance | Fast (DB lookup) | Fast (OS lookup) | Medium (API call) |
| Audit Trail | Yes | No | Yes |
| Rotation | Manual | Manual | Automated |
| Cost | Free | Free | Low-Medium |
| Complexity | Low | Low | Medium |
| Multi-environment | Difficult | Easy | Easy |
Connection Pool Metrics
| Metric | Recommended | Warning | Critical |
|---|---|---|---|
| Pool Utilization | < 70% | 70-90% | > 90% |
| Connection Wait Time | < 1s | 1-5s | > 5s |
| Connection Errors | < 1% | 1-5% | > 5% |
| Pool Overflow | < 50% | 50-80% | > 80% |
See Also
- Airflow Architecture â Core architecture including metadata database
- Operators and Hooks â Built-in operators that use hooks
- XCom Communications â Data passing between tasks
- Security Best Practices â Secrets management and security