Kubernetes Executor Deep Dive

Architecture Diagram

Formal Definitions

Detailed Explanation

What is the Kubernetes Executor?

The Kubernetes executor creates a separate pod for each task in your DAG. This provides complete isolation between tasks and dynamic resource allocation.

Key Insight: Each task gets its own container with dedicated CPU, memory, and network resources. No task can interfere with another task's execution.

How It Works

• Task Submission: Scheduler tells the K8s executor to run a task
• Pod Creation: Executor creates a new pod with the task's requirements
• Task Execution: Pod runs the task in isolation
• Cleanup: Pod is deleted after task completion (success or failure)

Basic K8s Executor Configuration

Pod Template Configuration

# pod_templates/default_template.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: airflow-worker
    component: worker
spec:
  serviceAccountName: airflow-worker
  containers:
    - name: base
      image: apache/airflow:2.8.0
      command:
        - "airflow"
        - "serve-logs"
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: 1000m
          memory: 2Gi
      env:
        - name: AIRFLOW__CORE__EXECUTOR
          value: "KubernetesExecutor"
        - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
          valueFrom:
            secretKeyRef:
              name: airflow-secrets
              key: database-url
      volumeMounts:
        - name: airflow-config
          mountPath: /opt/airflow/airflow.cfg
          subPath: airflow.cfg
        - name: dags
          mountPath: /opt/airflow/dags
      securityContext:
        runAsUser: 50000
        runAsGroup: 0
        fsGroup: 0
  volumes:
    - name: airflow-config
      configMap:
        name: airflow-config
    - name: dags
      persistentVolumeClaim:
        claimName: airflow-dags-pvc
  nodeSelector:
    node-type: worker
  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "worker"
      effect: "NoSchedule"
  restartPolicy: Never

Custom Pod Templates per Task

Pod Lifecycle States

Key Concepts Table

Code Examples

Advanced K8s Executor Configuration

Pod Monitoring and Cleanup

Resource-Aware Scheduling

Performance Metrics

K8s Executor vs Other Executors

Pod Resource Optimization

Best Practices for K8s Executor

1. Set resource requests and limits to prevent pods from consuming excessive cluster resources
2. Use pod templates to customize worker configurations for different workload types
3. Enable pod cleanup (delete_worker_pods = True) to prevent resource leaks
4. Configure node selectors to place pods on appropriate node pools
5. Use service accounts with minimal RBAC permissions for security
6. Monitor pod metrics to optimize resource allocation and costs

See Also

• Executors Comparison — Comparing all executor types
• Performance Tuning — Optimizing executor performance
• Monitoring and Alerting — Monitoring pod metrics
• Multi-Tenancy — Isolating workloads with K8s