System Design — Infrastructure
Load Balancing
Load balancing distributes incoming traffic across multiple servers to ensure high availability, reliability, and optimal resource utilization. It's one of the most critical components in any scalable system.
- Algorithms — Round-robin, least connections, consistent hashing
- L4 vs L7 — Transport vs application layer balancing
- Health Checks — Detecting and removing unhealthy servers
If you have more than one server, you need a load balancer.
What Is Load Balancing?
Load Balancer Placement
Load Balancing Algorithms
Round Robin
Distributes requests sequentially across servers.
Pros: Simple, even distribution for uniform workloads Cons: Ignores server capacity and current load
Weighted Round Robin
Assigns weights proportional to server capacity.
Least Connections
Routes to the server with fewest active connections.
Pros: Adapts to varying request durations Cons: Requires tracking connection state, doesn't account for connection weight
Consistent Hashing
Maps servers and requests to a hash ring, minimizing redistribution when servers are added or removed.
Pros: Minimal redistribution on scaling, works well with caches Cons: May be uneven without virtual nodes
Least Response Time
Routes to the server with the lowest average response time and fewest active connections.
L4 vs L7 Load Balancing
| Feature | L4 | L7 |
|---|---|---|
| OSI Layer | Transport (TCP/UDP) | Application (HTTP) |
| Routing basis | IP, port | URL, headers, cookies |
| Throughput | Higher (no payload inspection) | Lower (full inspection) |
| SSL termination | Can terminate | Typically terminates |
| Content-based routing | No | Yes (URL, header) |
| WebSocket support | Yes (pass-through) | Yes (with upgrade) |
| Use case | High-throughput, simple routing | Content routing, SSL offload |
Health Checks
Load balancers must detect and remove unhealthy servers.
Health Check Types
| Type | Description | Granularity |
|---|---|---|
| TCP | Can establish TCP connection | Port-level |
| HTTP | HTTP 200 response from health endpoint | Application-level |
| gRPC | gRPC health check protocol | Service-level |
| Custom | Application-specific check | Business-level |
Health Check Configuration
Health Check Parameters:
- Interval: 10 seconds (how often to check)
- Timeout: 5 seconds (max wait per check)
- Healthy threshold: 2 (consecutive successes to mark healthy)
- Unhealthy threshold: 3 (consecutive failures to mark unhealthy)
Global Server Load Balancing (GSLB)
For multi-region deployments, GSLB distributes traffic across data centers.
GSLB Strategies
| Strategy | How It Works |
|---|---|
| GeoDNS | Resolves to nearest data center by IP geolocation |
| Anycast | Multiple data centers share same IP; BGP routes to nearest |
| Latency-based | Routes to data center with lowest measured latency |
| Weighted | Distributes traffic by configurable weights |
Practice Exercises
-
Design: Design a load balancing strategy for a real-time gaming platform with 100M concurrent users. Consider: sticky sessions, geographic distribution, failover, and latency requirements.
-
Algorithm Selection: You have 5 servers with different capacities (2x, 1x, 1x, 1x, 0.5x). Which load balancing algorithm would you use? How would you configure it?
-
Architecture: Compare L4 and L7 load balancing for: (a) a TCP-based database proxy, (b) an API gateway with content-based routing, (c) a WebSocket chat service.
-
Troubleshooting: A load balancer is distributing traffic unevenly—server 1 gets 40% while others get 20% each. What could cause this? How would you diagnose and fix it?
What to Learn Next
-> Message Queues Kafka, RabbitMQ, event-driven architecture.
-> Microservices Service decomposition, discovery, and API gateways.
-> CAP Theorem Consistency models, availability, and partition tolerance.
-> Scalability Fundamentals Vertical vs horizontal scaling and capacity planning.
-> Databases SQL vs NoSQL, indexing, replication, and sharding.
-> API Design REST, GraphQL, gRPC, versioning, and rate limiting.