System Design Problems
Design a Real-time Analytics System
A real-time analytics system ingests high-volume event streams and provides sub-second query responses for dashboards, A/B testing, and business intelligence. Systems like Apache Kafka, Flink, and ClickPower enable real-time data analysis at massive scale.
- Stream Processing â Process millions of events per second in real-time
- Sub-second Queries â Dashboard responses in < 1 second
- Time-series Optimization â Data partitioned by time for efficient range queries
The challenge is balancing write throughput (ingest millions of events/sec) with read latency (query results in < 1 second) across time-partitioned data.
Requirements
Functional Requirements
- Ingest events from multiple sources (clickstream, IoT, logs)
- Store events with timestamps for time-range queries
- Support real-time aggregations (counts, sums, averages, percentiles)
- Dashboard queries with < 1 second latency
- Support for ad-hoc queries on recent data (last 24 hours)
- Historical data available for batch queries (last 90 days)
Non-Functional Requirements
- Write Throughput: 1M events/second
- Query Latency: < 1 second for dashboard queries
- Data Retention: 90 days hot, 1 year cold
- Accuracy: Exact for counts, approximate for percentiles (HyperLogLog)
- Availability: 99.99%
Back-of-the-Envelope Estimation
High-Level Architecture
Detailed Design
Lambda Architecture
The Lambda architecture combines batch and stream processing:
| Layer | Purpose | Technology |
|---|---|---|
| Batch Layer | Complete, accurate results over all data | Spark, Hive |
| Speed Layer | Low-latency results over recent data | Flink, Kafka Streams |
| Serving Layer | Merged view for queries | ClickHouse, Druid |
Pre-aggregation Strategy
Pre-aggregate data at write time to enable fast queries:
OLAP Database (ClickHouse)
Time-series Partitioning
Partition data by time for efficient range queries:
Approximate Algorithms
For high-cardinality analytics, use approximate algorithms:
| Algorithm | Purpose | Error | Memory |
|---|---|---|---|
| HyperLogLog | Distinct count | ~0.8% | 12 KB |
| Count-Min Sketch | Frequency estimation | Varies | Fixed |
| T-Digest | Percentile estimation | < 1% | 100 KB |
Practice Exercises
-
Design: How would you implement a real-time dashboard that shows the number of active users in the last 5 minutes? What data structure would you use?
-
Scale: If the system ingests 1M events/second, estimate the Kafka partition count needed and the ClickHouse cluster size for 90-day retention.
-
Consistency: In a Lambda architecture, how do you handle the transition from batch-computed results to stream-computed results? Design a merge strategy.
-
Optimization: How would you optimize ClickHouse queries for a dashboard that queries 10 different time ranges and 5 different dimensions simultaneously?
What to Learn Next
-> Design Metrics Monitoring Collecting and querying system metrics at scale.
-> Message Queues Kafka for event streaming and partitioning strategies.
-> Event-Driven Architecture Event sourcing and CQRS patterns.
-> Databases Column-oriented vs row-oriented storage trade-offs.
-> Caching Strategies Caching pre-aggregated results for dashboard queries.
-> Design Realtime Analytics This article (stream processing deep dive).