Data Systems
Kafka Deep Dive
Apache Kafka is the de facto standard for event streaming. Master its architecture, partitioning model, consumer groups, and exactly-once semantics for building event-driven systems.
- Distributed — Horizontally scalable across many brokers
- Durable — Persistent log with configurable retention
- Real-Time — Sub-millisecond latency for event delivery
Kafka is not just a message queue—it's a distributed event log.
Kafka Architecture
Key Concepts
| Concept | Description |
|---|---|
| Topic | Named stream of events (like a table) |
| Partition | Unit of parallelism within a topic |
| Offset | Unique sequential ID for each event in a partition |
| Broker | Kafka server that stores and serves data |
| Replica | Copy of a partition for fault tolerance |
| Consumer Group | Group of consumers that divide partition consumption |
Partitioning Model
Consumer Groups
| Configuration | Effect |
|---|---|
| N consumers = N partitions | Each consumer gets one partition |
| N consumers < N partitions | Some consumers get multiple partitions |
| N consumers > N partitions | Some consumers are idle |
| Rebalancing | Automatic redistribution when consumers join/leave |
Exactly-Once Semantics
| Mechanism | Purpose |
|---|---|
| Idempotent producer | Prevents duplicate writes during retries |
| Transactions | Atomic writes across multiple partitions |
| Consumer offset in transaction | Commit offset with processed data atomically |
| Transactional outbox | Atomic database write + event publish |
Kafka Retention
| Retention Type | Description |
|---|---|
| Time-based | Delete events after N days (default: 7 days) |
| Size-based | Keep only the last N GB per partition |
| Log compaction | Keep only the latest value per key |
| Infinite | Never delete (requires sufficient disk) |
Practice Exercises
-
Topic Design: Design the Kafka topics for an e-commerce order system. What topics would you create, how many partitions, and what retention policy?
-
Consumer Design: Design a consumer group for processing payment events. How do you ensure exactly-once processing when writing to a PostgreSQL database?
-
Partitioning Strategy: For a topic with user events, choose a partitioning strategy that ensures events for the same user are ordered but load is balanced. What happens when a user has significantly more events than others?
-
Architecture Decision: Compare Kafka with RabbitMQ for a task queue system. What are the trade-offs in terms of ordering, throughput, and replay capability?
What to Learn Next
-> Stream Processing Real-time data processing with Flink, Spark Streaming, and Kafka Streams.
-> Redis Deep Dive Redis data structures, persistence, clustering, and use cases.
-> Event-Driven Architecture Event sourcing, CQRS, and message-driven systems.
-> Message Queues Async processing, event-driven architecture, and pub/sub patterns.
-> Data Lake Architecture Storage, processing, and governance for large-scale data.
-> Batch Processing MapReduce, Spark, and distributed batch processing.