System Design Problems
Design a Metrics Monitoring System
A metrics monitoring system collects, stores, and visualizes time-series data for infrastructure and application monitoring. Systems like Prometheus, Datadog, and Grafana handle billions of data points daily, enabling operators to detect anomalies and alert on issues in real-time.
- Time-series Storage â Efficiently store metrics indexed by time and labels
- Real-time Ingestion â Process millions of data points per second
- Alerting â Detect anomalies and notify on-call engineers
Time-series data is append-heavy and read-heavy for recent ranges. The storage engine must optimize for sequential writes and range queries over time windows.
Requirements
Functional Requirements
- Ingest metrics from multiple sources (servers, containers, applications)
- Store time-series data (metric name, labels, timestamp, value)
- Query metrics over time ranges (last 5 min, 1 hour, 7 days)
- Aggregate metrics (rate, avg, sum, count, percentiles)
- Dashboard visualization with auto-refresh
- Alert rules that trigger notifications
- Metric discovery and labeling
Non-Functional Requirements
- Write Throughput: 10M data points/second
- Query Latency: < 1 second for dashboard queries
- Retention: 15 days raw, 1 year downsampled
- Availability: 99.99%
- Accuracy: Exact for counters, approximate for high-cardinality
Back-of-the-Envelope Estimation
Data Model
metric: http_requests_total
labels: {method: "GET", status: "200", endpoint: "/api/v1/users"}
timestamp: 1687267200000 (ms)
value: 15234
# This is one data point in one time series
# A system with 10K endpoints à 3 methods à 5 status codes = 150K time series
High-Level Architecture
Detailed Design
Time-Series Database (TSDB)
Storage Engine
Downsampling
Reduce storage by aggregating old data at lower resolution:
| Retention | Resolution | Data Points/Day |
|---|---|---|
| 0-15 days | 1 second | 86,400 |
| 15-90 days | 1 minute | 1,440 |
| 90-365 days | 1 hour | 24 |
| 1+ year | 1 day | 1 |
Alerting System
alert_rule: {
name: "High Error Rate",
query: "rate(http_requests_total{status="5xx"}[5m]) / rate(http_requests_total[5m])",
condition: "> 0.05", // > 5% error rate
duration: "5m", // Must be true for 5 minutes
severity: "critical",
notify: ["pagerduty", "slack"]
}
PromQL-style Query Language
Practice Exercises
-
Design: How would you implement a label index that supports efficient multi-label queries (e.g.,
method="GET" AND status="200" AND host=~"web-.*")? -
Scale: If the system ingests 10M data points/second with 15-day retention, estimate the storage needed with 10:1 compression and downsampling to 1-minute resolution after 15 days.
-
Alerting: Design an alert deduplication and grouping system that sends one notification for related alerts (e.g., 50 servers all reporting high CPU).
-
Optimization: How would you optimize dashboard queries that render 50 panels, each querying 10 time series over the last 24 hours?
What to Learn Next
-> Design Realtime Analytics Stream processing for real-time event analytics.
-> Observability Logs, metrics, and traces for system observability.
-> Databases Time-series databases and LSM-tree storage engines.
-> Message Queues Kafka for metric ingestion and streaming.
-> Design Notification System Alert notification delivery via multiple channels.
-> Caching Strategies Caching dashboard queries and metric aggregates.