🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
NEWSLIVESearch All Content
đŸ’ŧ Servicesâ„šī¸ Aboutâœ‰ī¸ ContactView Pricing Plansfrom $10

Security Best Practices in Apache Airflow

đŸŸĸ Free Lesson

Advertisement

Security Best Practices

Security LayersAuthenticationOAuth, LDAP, SAMLRBACRole-based accessEncryptionTLS, AES-256NetworkFirewalls, VPNSecrets BackendVault, AWS SMSecrets PriorityVault {'>'} AWS SM {'>'} GCP SM {'>'} Env Vars {'>'} DBComplianceGDPR, SOC2, HIPAA controlsAudit LoggingActivity trackingDefense in depth: multiple security layers protect against different threat vectors

Architecture Diagram

Formal Definitions

Detailed Explanation

RBAC Configuration

Role-Based Access Control restricts user permissions.


Default Roles:

RolePermissions
AdminFull access to all resources
OpCreate/edit DAGs, manage connections/variables
UserRead-only access to DAGs and history
ViewerRead-only access to DAGs

Custom Roles:

python
ROLES = {
    'DataEngineer': [
        (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
        (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_DAG),
    ],
    'DataAnalyst': [
        (permissions.ACTION_CAN_READ, permissions.RESOURCE_DAG),
    ],
}

OAuth Integration: Google, GitHub, LDAP, and custom providers supported.


Secrets Backend Setup

Store credentials in external secret management systems.


Supported Backends:

BackendProviderConfiguration
VaultHashiCorpbackend = airflow.providers.hashicorp.secrets.vault.VaultBackend
AWS SMAWSbackend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
GCP SMGoogle Cloudbackend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
python
# airflow.cfg
[secrets]
backend = airflow.providers.hashicorp.secrets.vault.VaultBackend
backend_kwargs = {
    "connections_path": "airflow/connections",
    "variables_path": "airflow/variables",
}

Note: Hooks automatically resolve connections from secrets backend.


Connection Encryption

Encrypt connections in transit and at rest.


Security Checklist:

PracticeImplementation
SSL/TLSsslmode: require in connection extras
FernetEnable AIRFLOW__CORE__FERNET_KEY for metadata encryption
HTTPSUse HTTPS for web server
NetworkDeploy in private VPC/subnet
python
# Encrypted connection example
conn = Connection(
    conn_id='production_db',
    conn_type='postgres',
    host='db.example.com',
    extra='{"sslmode": "require", "sslrootcert": "/path/to/ca.pem"}',
)

# Additional SSL parameters for strict verification
conn_params = {
    'sslmode': 'require',  # Enforce SSL
    'sslcert': '/path/to/client-cert.pem',
    'sslkey': '/path/to/client-key.pem',
    'sslrootcert': '/path/to/ca-cert.pem',
}

def verify_encryption():
    """Verify connection is encrypted."""
    hook = PostgresHook(postgres_conn_id='encrypted_db')

    result = hook.get_first("""
        SELECT
            pgssl.ssl_is_used as ssl_enabled,
            pgssl.ssl_version as ssl_version,
            pgssl.ssl_cipher as cipher
        FROM pg_stat_ssl pgssl
        JOIN pg_stat_activity pgact ON pgssl.pid = pgact.pid
        WHERE pgact.usename = current_user
    """)

    return {
        'ssl_enabled': result[0],
        'ssl_version': result[1],
        'cipher': result[2],
    }

Key Concepts Table

Security LayerComponentImplementationPriority
AuthenticationWebserverOAuth, LDAP, SAMLP0
AuthorizationRBACRole-based permissionsP0
EncryptionDatabaseTLS, AES-256P0
SecretsBackendVault, AWS SMP0
NetworkInfrastructureFirewalls, VPNP1
AuditLoggingActivity logsP1
CompliancePoliciesSOC2, GDPRP2

Code Examples

Security Audit Script

python
# security_audit.py
from airflow import settings
from airflow.models import Connection, Variable
from sqlalchemy import text
import json

def audit_security_posture():
    """Comprehensive security audit of Airflow deployment."""
    session = settings.Session()
    findings = []
    
    # Check for plaintext passwords in metadata DB
    plaintext_conns = session.query(Connection).filter(
        Connection.conn_type.notin_(['aws', 'google_cloud_platform'])
    ).all()
    
    for conn in plaintext_conns:
        if conn.password and not conn.password.startswith('{'):
            findings.append({
                'severity': 'HIGH',
                'category': 'Secrets',
                'finding': f'Plaintext password in connection: {conn.conn_id}',
                'recommendation': 'Move to secrets backend',
            })
    
    # Check for variables with sensitive data
    sensitive_patterns = ['password', 'secret', 'key', 'token']
    variables = session.query(Variable).all()
    
    for var in variables:
        if any(pattern in var.key.lower() for pattern in sensitive_patterns):
            findings.append({
                'severity': 'MEDIUM',
                'category': 'Secrets',
                'finding': f'Potentially sensitive variable: {var.key}',
                'recommendation': 'Move to secrets backend',
            })
    
    # Check for DAGs with hardcoded secrets
    import ast
    import os
    
    dags_folder = '/opt/airflow/dags'
    for root, dirs, files in os.walk(dags_folder):
        for file in files:
            if file.endswith('.py'):
                filepath = os.path.join(root, file)
                with open(filepath, 'r') as f:
                    content = f.read()
                
                # Simple pattern matching
                for pattern in ['password=', 'secret=', 'api_key=']:
                    if pattern in content:
                        findings.append({
                            'severity': 'HIGH',
                            'category': 'Code',
                            'finding': f'Potential hardcoded secret in {filepath}',
                            'recommendation': 'Use variables or secrets backend',
                        })
    
    return findings

def generate_security_report(findings):
    """Generate security audit report."""
    report = {
        'total_findings': len(findings),
        'high_severity': len([f for f in findings if f['severity'] == 'HIGH']),
        'medium_severity': len([f for f in findings if f['severity'] == 'MEDIUM']),
        'low_severity': len([f for f in findings if f['severity'] == 'LOW']),
        'findings': findings,
    }
    
    print(json.dumps(report, indent=2))
    return report

if __name__ == "__main__":
    findings = audit_security_posture()
    generate_security_report(findings)

Network Security Configuration

# docker-compose-security.yml
version: '3.8'

services:
  airflow-webserver:
    image: apache/airflow:2.8.0
    command: webserver
    networks:
      - airflow-internal
    environment:
      - AIRFLOW__CORE__FERNET_KEY=${FERNET_KEY}
      - AIRFLOW__WEBSERVER__EXPOSE_CONFIG=False
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2G
    # Security context
    security_opt:
      - no-new-privileges:true
    read_only: true
    tmpfs:
      - /tmp

  airflow-scheduler:
    image: apache/airflow:2.8.0
    command: scheduler
    networks:
      - airflow-internal
    environment:
      - AIRFLOW__CORE__FERNET_KEY=${FERNET_KEY}
    security_opt:
      - no-new-privileges:true

  postgres:
    image: postgres:15
    networks:
      - airflow-internal
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    # Disable network access except from Airflow
    # Use internal network only

  vault:
    image: hashicorp/vault:1.15
    networks:
      - airflow-internal
    cap_add:
      - IPC_LOCK
    environment:
      VAULT_ADDR: 'http://0.0.0.0:8200'
    volumes:
      - vault_data:/vault/file
    command: server -dev

networks:
  airflow-internal:
    driver: bridge
    internal: true  # No external access

secrets:
  db_password:
    file: ./secrets/db_password.txt

volumes:
  postgres_data:
  vault_data:

Performance Metrics

Security Posture Score

MetricScoreWeightStatus
Secrets in Backend100%30%PASS
TLS Enabled100%25%PASS
RBAC Configured80%20%PASS
Audit Logging90%15%PASS
Network Segmentation70%10%WARNING

Compliance Checklist

RequirementStatusEvidenceOwner
SOC2 - Access ControlsPASSRBAC configuredSecurity
SOC2 - EncryptionPASSTLS 1.3 enabledPlatform
GDPR - Data ProtectionPASSEncryption at restPlatform
GDPR - Audit TrailPASSLogging enabledSecurity
HIPAA - PHI HandlingN/ANo PHI processed-

See Also

—
☆☆☆☆☆
0 ratings

Rate & Feedback

Need Expert Airflow Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement