System Design - Infrastructure
Message Queues
Message queues decouple producers from consumers, enabling asynchronous communication between services. They are the backbone of scalable, resilient, event-driven architectures.
- Kafka - Distributed event streaming platform for high-throughput data
- RabbitMQ - Traditional message broker with flexible routing
- Event-Driven - Architecture pattern where events drive system behavior
The best distributed system is the one where components do not need to know about each other.
What Are Message Queues?
Why Use Message Queues?
Benefits of Message Queues
| Benefit | Description |
|---|---|
| Decoupling | Producer and consumer do not need to know about each other |
| Asynchronous | Producer does not wait for consumer to process |
| Buffering | Queue absorbs traffic spikes, protecting downstream services |
| Scalability | Add consumers independently of producers |
| Resilience | If consumer fails, messages persist in queue |
| Ordering | Maintains message order within a partition or queue |
Kafka vs RabbitMQ
| Feature | Kafka | RabbitMQ |
|---|---|---|
| Model | Distributed log | Message broker |
| Storage | Durable, append-only log | In-memory (persistent optional) |
| Message retention | Configurable (time/size) | Deleted after acknowledgment |
| Consumer model | Pull (consumers poll) | Push (broker delivers) |
| Ordering | Guaranteed within partition | Guaranteed within queue |
| Throughput | Millions of msgs/sec | Tens of thousands/sec |
| Latency | Milliseconds | Microseconds |
| Protocol | Custom binary | AMQP, MQTT, STOMP |
| Best for | Event streaming, data pipelines, logging | Task queues, RPC, complex routing |
Kafka Architecture
Messaging Patterns
Point-to-Point (Queue)
Publish-Subscribe (Pub/Sub)
Event Sourcing
Message Delivery Guarantees
| Guarantee | Description | Implementation |
|---|---|---|
| At-most-once | Message may be lost, never duplicated | Fire and forget, no ack |
| At-least-once | Message may be duplicated, never lost | Ack after processing, retry |
| Exactly-once | Message delivered exactly once | Idempotent producers, transactional consumers |
Dead Letter Queues
When a message cannot be processed after a configured number of retries, it is moved to a Dead Letter Queue (DLQ) for investigation.
Practice Exercises
-
Design: Design an event-driven order processing system for an e-commerce platform. Orders flow through: payment, inventory check, shipping, and notification. Use Kafka or RabbitMQ and justify your choice.
-
Trade-offs: Compare Kafka and RabbitMQ for: (a) real-time log aggregation, (b) background job processing, (c) IoT sensor data ingestion. Which would you choose for each and why?
-
Architecture: Design a system that processes 1 million events per second with exactly-once delivery semantics. What components would you need? What are the failure modes?
-
Analysis: Your RabbitMQ queue is accumulating messages faster than consumers can process them. What are 5 strategies to handle this situation?
What to Learn Next
-> Microservices Service decomposition, discovery, and API gateways.
-> CAP Theorem Consistency models, availability, and partition tolerance.
-> Load Balancing Algorithms, health checks, and L4 vs L7.
-> Databases SQL vs NoSQL, indexing, replication, and sharding.
-> Caching Strategies Redis, Memcached, cache invalidation, and write strategies.
-> API Design REST, GraphQL, gRPC, versioning, and rate limiting.