Operations
Observability
Observability is the ability to understand a system's internal state from its external outputs. In distributed systems, observability is essential for debugging, performance tuning, and maintaining reliability.
- Logs β Discrete events with context
- Metrics β Numerical measurements over time
- Traces β Request path across service boundaries
You can't fix what you can't see.
What Is Observability?
The three pillars of observability provide complementary views into system behavior.
Three Pillars
| Pillar | What It Tells You | Example |
|---|---|---|
| Logs | What happened at a point in time | "User 123 failed to login at 10:00:01" |
| Metrics | How the system is performing over time | "Request rate: 1000 QPS, error rate: 0.1%" |
| Traces | Where time was spent in a request | "API gateway: 5ms, Auth: 10ms, DB: 50ms" |
Logging
Structured logging enables efficient querying and analysis.
Structured log example:
{
"timestamp": "2024-01-15T10:00:01.123Z",
"level": "ERROR",
"service": "payment-service",
"trace_id": "abc123def456",
"user_id": "789",
"message": "Payment processing failed",
"error": "Insufficient funds",
"amount": 29.99,
"currency": "USD"
}
Log Levels
| Level | When to Use |
|---|---|
| DEBUG | Detailed diagnostic information |
| INFO | Normal operation milestones |
| WARN | Unexpected but recoverable conditions |
| ERROR | Failures requiring attention |
| FATAL | System cannot continue running |
Metrics
Numerical measurements aggregated over time.
Four Golden Signals
Google's SRE team identifies four key signals for monitoring:
| Signal | Description | Example |
|---|---|---|
| Latency | Time to serve a request | p99 = 200ms |
| Traffic | Demand on the system | 10,000 QPS |
| Errors | Rate of failed requests | 0.1% error rate |
| Saturation | How full the system is | 70% CPU utilization |
RED Method
For request-driven services:
Prometheus
The standard for metrics collection and alerting.
Prometheus Architecture
Distributed Tracing
Tracking requests across service boundaries.
Tracing Concepts
| Concept | Description |
|---|---|
| Trace | Complete journey of a request through the system |
| Span | A single unit of work within a trace |
| Trace ID | Unique identifier linking all spans in a trace |
| Parent Span ID | Reference to the span that initiated this span |
| Context Propagation | Passing trace context between services |
OpenTelemetry
Alerting
Practice Exercises
-
Design: Design an observability stack for a microservices application with 20 services. Include logging, metrics, tracing, and alerting strategies.
-
Metrics: Write PromQL queries for: request rate by service, p99 latency, error rate, and saturation (CPU utilization).
-
Tracing: Draw a trace diagram for an e-commerce checkout: API Gateway β Order Service β Payment Service β Inventory Service. Where would you expect bottlenecks?
-
Alerting: Design alerting rules for an API gateway. Include rules for error rate, latency, and saturation with appropriate thresholds and durations.
What to Learn Next
-> Service Mesh Envoy, Istio, and sidecar proxy patterns.
-> CI/CD Pipelines Continuous integration and deployment strategies.
-> Containerization Docker, Kubernetes, and pod scheduling.
-> Security Patterns Authentication, authorization, encryption, and mTLS.
-> Cost Optimization Cloud cost management and right-sizing.
-> Load Balancing Distribution algorithms and health checks.