LLM Production
LLM Monitoring and Observability β Visibility Into AI Systems
Effective monitoring for LLMs goes beyond traditional APM. You need to track token usage, semantic drift, hallucination rates, and output quality in real time.
- Three Pillars β Logs, metrics, and traces for LLM systems
- Drift Detection β Monitoring model behavior degradation over time
- Quality Metrics β Hallucination detection, toxicity monitoring, accuracy tracking
You cannot improve what you cannot measure.
LLM Monitoring and Observability
Production LLM systems require specialized observability that captures both system-level metrics (latency, throughput, error rates) and model-level metrics (output quality, semantic consistency, drift). Traditional monitoring approaches are insufficient for the unique failure modes of LLMs.
The Three Pillars of LLM Observability
1. Structured Logging
Every LLM interaction should produce structured logs capturing the full context.
Essential Log Fields:
request_id: Unique identifier for tracingmodel_version: Exact model checkpoint usedinput_tokens/output_tokens: Token countsttft: Time to first token (milliseconds)tpot: Time per output tokentotal_latency: End-to-end response timefinish_reason: Stop reason (stop token, length, safety)quality_score: Automated quality assessment
2. Distributed Tracing
LLM pipelines often involve multiple stages: prompt construction, retrieval (RAG), inference, post-processing, and safety checks.
3. Quantitative Metrics
Key Metrics to Track
System Metrics
| Metric | Description | Alert Threshold |
|---|---|---|
| TTFT (p95) | Time to first token | > 500ms |
| TPOT (p95) | Time per output token | > 50ms |
| Throughput | Tokens per second | < 80% capacity |
| Error Rate | Failed requests | > 1% |
| GPU Utilization | SM active percentage | < 60% or > 95% |
Quality Metrics
| Metric | Description | Alert Threshold |
|---|---|---|
| Hallucination Rate | Factual inconsistencies | > 5% |
| Toxicity Score | Harmful content detection | > 0.1 |
| Relevance Score | Query-response alignment | < 0.7 |
| Refusal Rate | Model declining to answer | > 20% |
| Format Compliance | Structured output adherence | < 95% |
Drift Detection
Semantic Drift
Output Distribution Drift
Monitor changes in token probability distributions to detect model behavior shifts.
Alerting Strategies
Multi-Tier Alerting
Practice Exercises
-
Conceptual: Explain why traditional APM metrics (CPU, memory, request count) are insufficient for monitoring LLM systems. What additional metrics are needed?
-
Mathematical: Given two output distributions P_t1 and P_t2 with vocabulary size 50,000, calculate the KL divergence if the distributions differ by 0.1% in probability mass shifted uniformly across 100 tokens.
-
Practical: Design a monitoring dashboard for an RAG-based LLM application that includes retrieval quality metrics, generation quality metrics, and system performance metrics.
-
Research: Compare supervised drift detection methods (requiring labeled data) versus unsupervised methods (using statistical tests) for LLM output monitoring.
What to Learn Next
-> LLM Serving Architectures vLLM, TGI, TensorRT-LLM, and serving patterns for production deployments.
-> AB Testing for LLMs Experiment design, statistical significance, and canary deployments.
-> LLM Evaluation in Production Online evaluation, user feedback loops, and quality assurance.
-> LLM Disaster Recovery Failover, backup models, and graceful degradation strategies.
-> Cost Optimization for LLMs Token economics, caching, and batching for cost efficiency.
-> LLM Security Best Practices Protecting systems from prompt injection and adversarial attacks.