🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
đŸ’ŧ Servicesâ„šī¸ Aboutâœ‰ī¸ ContactView Pricing Plansfrom $10

Security Best Practices in Apache Airflow

đŸŸĸ Free Lesson

Advertisement

Security Best Practices

Security LayersAuthenticationOAuth, LDAP, SAMLRBACRole-based accessEncryptionTLS, AES-256NetworkFirewalls, VPNSecrets BackendVault, AWS SMSecrets PriorityVault {'>'} AWS SM {'>'} GCP SM {'>'} Env Vars {'>'} DBComplianceGDPR, SOC2, HIPAA controlsAudit LoggingActivity trackingDefense in depth: multiple security layers protect against different threat vectors

Architecture Diagram

Formal Definitions

Detailed Explanation

RBAC Configuration

Role-Based Access Control restricts user permissions.


Default Roles:

RolePermissions
AdminFull access to all resources
OpCreate/edit DAGs, manage connections/variables
UserRead-only access to DAGs and history
ViewerRead-only access to DAGs

Custom Roles:

ROLES = {
    'DataEngineer': [
        (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
        (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG),
    ],
    'DataAnalyst': [
        (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
    ],
}

OAuth Integration: Google, GitHub, LDAP, and custom providers supported.


Secrets Backend Setup

Store credentials in external secret management systems.


Supported Backends:

BackendProviderConfiguration
VaultHashiCorpbackend = airflow.providers.hashicorp.secrets.vault.VaultBackend
AWS SMAWSbackend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
GCP SMGoogle Cloudbackend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
# airflow.cfg
[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {
    "connections_path": "airflow/connections",
    "variables_path": "airflow/variables",
}

Note: Hooks automatically resolve connections from secrets backend.


Connection Encryption

Encrypt connections in transit and at rest.


Security Checklist:

PracticeImplementation
SSL/TLSsslmode: require in connection extras
FernetEnable AIRFLOW__CORE__FERNET_KEY for metadata encryption
HTTPSUse HTTPS for web server
NetworkDeploy in private VPC/subnet
# Encrypted connection example
conn = Connection(
    conn_id='production_db',
    conn_type='postgres',
    host='db.example.com',
    extra='{"sslmode": "require", "sslrootcert": "/path/to/ca.pem"}',
)

'sslmode': 'require', # Enforce SSL 'sslcert': '/path/to/client-cert.pem', 'sslkey': '/path/to/client-key.pem', 'sslrootcert': '/path/to/ca-cert.pem', }

return conn_params

def verify_encryption(): """Verify connection is encrypted.""" hook = PostgresHook(postgres_conn_id='encrypted_db')

result = hook.get_first(""" SELECT pgssl.ssl_is_used as ssl_enabled, pgssl.ssl_version as ssl_version, pgssl.ssl_cipher as cipher FROM pg_stat_ssl pgssl JOIN pg_stat_activity pgact ON pgssl.pid = pgact.pid WHERE pgact.usename = current_user """)

return { 'ssl_enabled': result[0], 'ssl_version': result[1], 'cipher': result[2], }

Architecture Diagram

<MathKeyFormula
  title="RBAC Permission Check"
  tex={`\\text{Access}(r, o, a) = \\begin{cases} \\text{granted} & \\text{if } (r, o, a) \\\\ & \\in P_{\\text{role}} \\\\ \\text{denied} & \\text{otherwise} \\end{cases}`}
/>

<MathFormula
  title="Secret Rotation Interval"
  tex={`T_{\\text{rotation}} \\leq \\min(T_{\\text{policy}}, T_{\\text{compromise\\_risk}})`}
/>

<MathNote type="info">
Never store secrets in DAG files or environment variables in plaintext. Use Airflow's Secrets Backend (Vault, AWS Secrets Manager, GCP Secret Manager) for all credentials.
</MathNote>

<MathNote type="tip">
Enable audit logging to track who accessed what secrets and when. This is required for SOC2 and GDPR compliance.
</MathNote>

## Key Concepts Table

| Security Layer | Component | Implementation | Priority |
|----------------|-----------|----------------|----------|
| **Authentication** | Webserver | OAuth, LDAP, SAML | P0 |
| **Authorization** | RBAC | Role-based permissions | P0 |
| **Encryption** | Database | TLS, AES-256 | P0 |
| **Secrets** | Backend | Vault, AWS SM | P0 |
| **Network** | Infrastructure | Firewalls, VPN | P1 |
| **Audit** | Logging | Activity logs | P1 |
| **Compliance** | Policies | SOC2, GDPR | P2 |

## Code Examples

### Security Audit Script

```python
# security_audit.py
from airflow import settings
from airflow.models import Connection, Variable
from sqlalchemy import text
import json

def audit_security_posture():
    """Comprehensive security audit of Airflow deployment."""
    session = settings.Session()
    findings = []
    
    # Check for plaintext passwords in metadata DB
    plaintext_conns = session.query(Connection).filter(
        Connection.conn_type.notin_(['aws', 'google_cloud_platform'])
    ).all()
    
    for conn in plaintext_conns:
        if conn.password and not conn.password.startswith('&#123;'):
            findings.append(&#123;
                'severity': 'HIGH',
                'category': 'Secrets',
                'finding': f'Plaintext password in connection: &#123;conn.conn_id&#125;',
                'recommendation': 'Move to secrets backend',
            &#125;)
    
    # Check for variables with sensitive data
    sensitive_patterns = ['password', 'secret', 'key', 'token']
    variables = session.query(Variable).all()
    
    for var in variables:
        if any(pattern in var.key.lower() for pattern in sensitive_patterns):
            findings.append(&#123;
                'severity': 'MEDIUM',
                'category': 'Secrets',
                'finding': f'Potentially sensitive variable: &#123;var.key&#125;',
                'recommendation': 'Move to secrets backend',
            &#125;)
    
    # Check for DAGs with hardcoded secrets
    import ast
    import os
    
    dags_folder = '/opt/airflow/dags'
    for root, dirs, files in os.walk(dags_folder):
        for file in files:
            if file.endswith('.py'):
                filepath = os.path.join(root, file)
                with open(filepath, 'r') as f:
                    content = f.read()
                
                # Simple pattern matching
                for pattern in ['password=', 'secret=', 'api_key=']:
                    if pattern in content:
                        findings.append(&#123;
                            'severity': 'HIGH',
                            'category': 'Code',
                            'finding': f'Potential hardcoded secret in &#123;filepath&#125;',
                            'recommendation': 'Use variables or secrets backend',
                        &#125;)
    
    return findings

def generate_security_report(findings):
    """Generate security audit report."""
    report = &#123;
        'total_findings': len(findings),
        'high_severity': len([f for f in findings if f['severity'] == 'HIGH']),
        'medium_severity': len([f for f in findings if f['severity'] == 'MEDIUM']),
        'low_severity': len([f for f in findings if f['severity'] == 'LOW']),
        'findings': findings,
    &#125;
    
    print(json.dumps(report, indent=2))
    return report

if __name__ == "__main__":
    findings = audit_security_posture()
    generate_security_report(findings)

Network Security Configuration

# docker-compose-security.yml
version: '3.8'

services:
  airflow-webserver:
    image: apache/airflow:2.8.0
    command: webserver
    networks:
      - airflow-internal
    environment:
      - AIRFLOW__CORE__FERNET_KEY=$&#123;FERNET_KEY&#125;
      - AIRFLOW__WEBSERVER__EXPOSE_CONFIG=False
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2G
    # Security context
    security_opt:
      - no-new-privileges:true
    read_only: true
    tmpfs:
      - /tmp

  airflow-scheduler:
    image: apache/airflow:2.8.0
    command: scheduler
    networks:
      - airflow-internal
    environment:
      - AIRFLOW__CORE__FERNET_KEY=$&#123;FERNET_KEY&#125;
    security_opt:
      - no-new-privileges:true

  postgres:
    image: postgres:15
    networks:
      - airflow-internal
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    # Disable network access except from Airflow
    # Use internal network only

  vault:
    image: hashicorp/vault:1.15
    networks:
      - airflow-internal
    cap_add:
      - IPC_LOCK
    environment:
      VAULT_ADDR: 'http://0.0.0.0:8200'
    volumes:
      - vault_data:/vault/file
    command: server -dev

networks:
  airflow-internal:
    driver: bridge
    internal: true  # No external access

secrets:
  db_password:
    file: ./secrets/db_password.txt

volumes:
  postgres_data:
  vault_data:

Performance Metrics

Security Posture Score

MetricScoreWeightStatus
Secrets in Backend100%30%PASS
TLS Enabled100%25%PASS
RBAC Configured80%20%PASS
Audit Logging90%15%PASS
Network Segmentation70%10%WARNING

Compliance Checklist

RequirementStatusEvidenceOwner
SOC2 - Access ControlsPASSRBAC configuredSecurity
SOC2 - EncryptionPASSTLS 1.3 enabledPlatform
GDPR - Data ProtectionPASSEncryption at restPlatform
GDPR - Audit TrailPASSLogging enabledSecurity
HIPAA - PHI HandlingN/ANo PHI processed-

See Also

Need Expert Airflow Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement