Kafka on K8s: Strimzi, Operators, Persistent Volumes

Difficulty: Senior | Asked at: Cloud-native companies, DevOps roles

ℹ️Interview Context

Running Kafka on Kubernetes is increasingly common. Interviewers expect you to understand the challenges, operator patterns, and how to manage stateful workloads on K8s.

The Question

How do you run Kafka on Kubernetes? Explain the role of operators like Strimzi, how persistent volumes work for stateful sets, and what operational challenges exist.

Kafka on K8s Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│              Kubernetes Cluster                           │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │            Strimzi Operator                      │   │
│  │  - Manages Kafka lifecycle                      │   │
│  │  - Creates StatefulSets, Services               │   │
│  │  - Handles upgrades and scaling                 │   │
│  └─────────────────────────────────────────────────┘   │
│                          │                               │
│                          ↓                               │
│  ┌─────────────────────────────────────────────────┐   │
│  │          Kafka StatefulSet                       │   │
│  │  ┌─────────┐  ┌─────────┐  ┌─────────┐        │   │
│  │  │ kafka-0 │  │ kafka-1 │  │ kafka-2 │        │   │
│  │  │ ┌─────┐ │  │ ┌─────┐ │  │ ┌─────┐ │        │   │
│  │  │ │ PVC │ │  │ │ PVC │ │  │ │ PVC │ │        │   │
│  │  │ └─────┘ │  │ └─────┘ │  │ └─────┘ │        │   │
│  │  └─────────┘  └─────────┘  └─────────┘        │   │
│  └─────────────────────────────────────────────────┘   │
│                          │                               │
│                          ↓                               │
│  ┌─────────────────────────────────────────────────┐   │
│  │          PersistentVolumeClaims                  │   │
│  │  kafka-data-kafka-0  kafka-data-kafka-1         │   │
│  │  kafka-data-kafka-2                             │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Strimzi Operator

Strimzi Architecture

strimzi_components = {
    'Cluster Operator': {
        'purpose': 'Manages Kafka, Connect, MirrorMaker, etc.',
        'scope': 'Namespace or cluster-wide',
        'features': [
            'Create/delete Kafka clusters',
            'Scale brokers up/down',
            'Handle rolling upgrades',
            'Manage TLS certificates'
        ]
    },
    'Topic Operator': {
        'purpose': 'Manages Kafka topics',
        'features': [
            'Create/delete topics via CRD',
            'Validate topic configurations',
            'Monitor topic health'
        ]
    },
    'User Operator': {
        'purpose': 'Manages Kafka users',
        'features': [
            'Create/delete users via CRD',
            'Manage SCRAM credentials',
            'Handle ACLs'
        ]
    }
}

Strimzi Kafka Cluster CRD

# kafka-cluster.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
spec:
  kafka:
    version: 3.6.1
    replicas: 3
    
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.6"
    
    storage:
      type: persistent-claim
      size: 100Gi
      class: fast-ssd
      deleteClaim: false
    
    resources:
      requests:
        memory: 4Gi
        cpu: 2
      limits:
        memory: 8Gi
        cpu: 4
    
    rack:
      topologyKey: topology.kubernetes.io/zone
    
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  
  zookeeper:
    replicas: 3
    
    storage:
      type: persistent-claim
      size: 20Gi
      class: fast-ssd
      deleteClaim: false
    
    resources:
      requests:
        memory: 2Gi
        cpu: 1
      limits:
        memory: 4Gi
        cpu: 2
  
  entityOperator:
    topicOperator: {}
    userOperator: {}

Strimzi Scaling

# Scale Kafka brokers
kubectl patch kafka my-cluster -n kafka --type merge -p '
{
  "spec": {
    "kafka": {
      "replicas": 5
    }
  }
}'

# Scale ZooKeeper
kubectl patch kafka my-cluster -n kafka --type merge -p '
{
  "spec": {
    "zookeeper": {
      "replicas": 5
    }
  }
}'

# Check status
kubectl get kafka my-cluster -n kafka -o yaml

Strimzi Rolling Upgrade

# Update Kafka version
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.7.0  # Upgrade from 3.6.1
    # ... rest of config

# Strimzi handles rolling upgrade automatically
# Monitor progress
kubectl get pods -n kafka -w

# Check upgrade status
kubectl get kafka my-cluster -n kafka -o jsonpath='{.status.conditions}'

ℹ️Strimzi Benefits

Strimzi provides:

Declarative Kafka management via CRDs
Automated rolling upgrades
Built-in TLS and authentication
Topic and user management
Integration with Kubernetes ecosystem

Persistent Volumes

PV/PVC Configuration

# StorageClass for Kafka
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iopsPerGB: "10"
  encrypted: "true"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

---
# PersistentVolumeClaim for Kafka broker
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kafka-data-kafka-0
  namespace: kafka
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi

Volume Configuration

pv_config = {
    'storage_classes': {
        'fast-ssd': {
            'provisioner': 'kubernetes.io/aws-ebs',
            'parameters': {'type': 'gp3'},
            'best_for': 'Production Kafka'
        },
        'standard': {
            'provisioner': 'kubernetes.io/aws-ebs',
            'parameters': {'type': 'gp2'},
            'best_for': 'Development/testing'
        },
        'local-ssd': {
            'provisioner': 'kubernetes.io/no-provisioner',
            'best_for': 'Maximum performance'
        }
    },
    'access_modes': {
        'ReadWriteOnce': 'Single node access (Kafka brokers)',
        'ReadWriteMany': 'Multi-node access (not recommended for Kafka)'
    },
    'reclaim_policies': {
        'Delete': 'Delete PVC when Pod is deleted',
        'Retain': 'Keep PVC when Pod is deleted (recommended)'
    }
}

Volume Performance

def estimate_volume_performance(
    storage_type: str,
    size_gb: int,
    iops: int
) -> dict:
    """
    Estimate volume performance for Kafka.
    """
    performance = {
        'gp2': {
            'throughput_mbps': 128 + (size_gb * 0.5),
            'iops': min(16000, 3 * size_gb),
            'latency_ms': 1
        },
        'gp3': {
            'throughput_mbps': 125 + (iops * 0.25),
            'iops': min(16000, iops),
            'latency_ms': 0.5
        },
        'io2': {
            'throughput_mbps': 1000,
            'iops': min(64000, iops),
            'latency_ms': 0.1
        },
        'st1': {
            'throughput_mbps': 500,
            'iops': 500,
            'latency_ms': 5
        }
    }
    
    return performance.get(storage_type, performance['gp3'])

# Example
perf = estimate_volume_performance('gp3', 100, 3000)
print(f"Throughput: {perf['throughput_mbps']} MB/s")
print(f"IOPS: {perf['iops']}")

⚠️Volume Warning

Kafka requires fast, persistent storage. Avoid:

Network-attached storage with high latency
EBS volumes smaller than 100GB (IOPS scaling)
Storage classes with reclaimPolicy=Delete

Resource Management

Resource Requests and Limits

# Kafka broker resources
resources:
  requests:
    memory: 4Gi
    cpu: "2"
    ephemeral-storage: 10Gi
  limits:
    memory: 8Gi
    cpu: "4"
    ephemeral-storage: 20Gi

# Resource formula:
# memory.requests = JVM heap + OS overhead (30%)
# memory.limits = 2x requests for burst capacity
# cpu.requests = Expected average usage
# cpu.limits = 2x requests for burst capacity

Resource Calculation

def calculate_kafka_resources(
    num_partitions: int,
    avg_message_size: int,
    messages_per_sec: int,
    replication_factor: int
) -> dict:
    """
    Calculate recommended Kafka resources.
    """
    # Memory calculation
    # JVM heap: 6GB recommended for production
    jvm_heap_gb = 6
    
    # OS page cache: 50% of remaining memory
    # Total memory: JVM heap / 0.7
    total_memory_gb = jvm_heap_gb / 0.7
    
    # CPU calculation
    # Base: 2 cores for Kafka broker
    # Additional: 0.5 core per 10,000 messages/sec
    base_cpu = 2
    additional_cpu = messages_per_sec / 10000 * 0.5
    total_cpu = base_cpu + additional_cpu
    
    # Storage calculation
    # Daily data: messages_per_sec * avg_message_size * 86400
    daily_data_gb = (messages_per_sec * avg_message_size * 86400) / (1024**3)
    
    return {
        'memory_requests': f'{int(total_memory_gb)}Gi',
        'memory_limits': f'{int(total_memory_gb * 2)}Gi',
        'cpu_requests': f'{int(total_cpu)}',
        'cpu_limits': f'{int(total_cpu * 2)}',
        'storage_per_broker': f'{int(daily_data_gb * 7 * replication_factor * 1.2)}Gi'
    }

# Example
resources = calculate_kafka_resources(
    num_partitions=12,
    avg_message_size=500,
    messages_per_sec=10000,
    replication_factor=3
)
print(f"Memory: {resources['memory_requests']} / {resources['memory_limits']}")
print(f"CPU: {resources['cpu_requests']} / {resources['cpu_limits']}")

Pod Disruption Budgets

# Pod Disruption Budget for Kafka
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
  namespace: kafka
spec:
  minAvailable: 2  # At least 2 brokers must be available
  selector:
    matchLabels:
      app: kafka
      strimzi.io/cluster: my-cluster

# Or use maxUnavailable
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
spec:
  maxUnavailable: 1  # At most 1 broker can be unavailable
  selector:
    matchLabels:
      app: kafka

ℹ️PDB Importance

Pod Disruption Budgets protect against voluntary disruptions (node drains, upgrades). For Kafka, ensure at least N-1 brokers are always available to maintain replication.

Network Configuration

Service Configuration

# Headless service for StatefulSet
apiVersion: v1
kind: Service
metadata:
  name: kafka-brokers
  namespace: kafka
spec:
  clusterIP: None
  selector:
    app: kafka
    strimzi.io/cluster: my-cluster
  ports:
    - name: plain
      port: 9092
    - name: tls
      port: 9093

---
# Bootstrap service
apiVersion: v1
kind: Service
metadata:
  name: kafka-bootstrap
  namespace: kafka
spec:
  selector:
    app: kafka
    strimzi.io/cluster: my-cluster
  ports:
    - name: plain
      port: 9092
    - name: tls
      port: 9093

Network Policies

# Network policy for Kafka
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: kafka-network-policy
  namespace: kafka
spec:
  podSelector:
    matchLabels:
      app: kafka
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: kafka-consumer
        - podSelector:
            matchLabels:
              app: kafka-producer
      ports:
        - port: 9092
        - port: 9093
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: kafka
      ports:
        - port: 9092
        - port: 9093

DNS Configuration

dns_config = {
    'headless_service': 'kafka-brokers.kafka.svc.cluster.local',
    'broker_dns': [
        'kafka-0.kafka-brokers.kafka.svc.cluster.local',
        'kafka-1.kafka-brokers.kafka.svc.cluster.local',
        'kafka-2.kafka-brokers.kafka.svc.cluster.local'
    ],
    'bootstrap_dns': 'kafka-bootstrap.kafka.svc.cluster.local',
    'client_config': {
        'bootstrap.servers': 'kafka-bootstrap:9092',
        'security.protocol': 'SSL',
        'ssl.truststore.location': '/etc/kafka/secrets/truststore.jks'
    }
}

Operational Challenges

Common Issues and Solutions

k8s_challenges = {
    'StatefulSet Scaling': {
        'issue': 'Kafka requires ordered, stable network IDs',
        'solution': 'Use StatefulSet with headless service',
        'implementation': 'Strimzi handles this automatically'
    },
    'Persistent Volume Management': {
        'issue': 'PVs must survive Pod restarts',
        'solution': 'Use Retain reclaim policy',
        'implementation': 'StorageClass with reclaimPolicy: Retain'
    },
    'Pod Disruptions': {
        'issue': 'Node drains can kill brokers',
        'solution': 'Pod Disruption Budgets + node affinity',
        'implementation': 'PDB with minAvailable=N-1'
    },
    'Resource Contention': {
        'issue': 'Noisy neighbor problems',
        'solution': 'Resource requests/limits + node affinity',
        'implementation': 'Dedicated node pools for Kafka'
    },
    'Network Partitions': {
        'issue': 'K8s network can partition',
        'solution': 'Anti-affinity rules + multiple AZs',
        'implementation': 'podAntiAffinity with topologySpreadConstraints'
    }
}

Pod Anti-Affinity

# Ensure Kafka brokers spread across nodes
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - kafka
              topologyKey: kubernetes.io/hostname
      
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: kafka
      
      tolerations:
        - key: "dedicated"
          operator: "Equal"
          value: "kafka"
          effect: "NoSchedule"

ℹ️Anti-Affinity

Use requiredDuringSchedulingIgnoredDuringExecution for production. It ensures brokers are on different nodes but allows scheduling to continue if nodes fail. Use topologySpreadConstraints for zone-aware distribution.

Monitoring on K8s

# Prometheus ServiceMonitor for Kafka
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kafka-metrics
  namespace: kafka
spec:
  selector:
    matchLabels:
      app: kafka
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

---
# Grafana dashboard ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-grafana-dashboard
  namespace: monitoring
data:
  kafka-dashboard.json: |
    {
      "dashboard": {
        "title": "Kafka on Kubernetes",
        "panels": [...]
      }
    }

⚠️Key Insight

Kafka on Kubernetes requires careful planning. The Strimzi operator significantly simplifies operations, but you still need to understand the underlying challenges. Always test in staging before production.