LLM Production
LLM Versioning and Rollouts β Safe Model Evolution
Managing model versions in production requires careful coordination of artifact storage, rollback capabilities, and gradual traffic shifting to minimize risk.
- Version Management β Model artifacts, metadata, and reproducibility
- Rollout Strategies β Canary, blue-green, and shadow deployments
- Rollback β Automated triggers and recovery procedures
Ship fast, but ship safely.
LLM Versioning and Rollouts
Deploying LLMs in production requires a systematic approach to versioning, testing, and rolling out changes. Unlike traditional software, model changes can have non-obvious behavioral impacts that only surface under real-world conditions.
Version Management
Model Artifacts
A complete LLM version includes:
Versioning Schema:
Semantic Versioning for Models
Rollout Strategies
Canary Deployment
Traffic Ramp Schedule:
| Stage | Traffic | Duration | Gate Criteria |
|---|---|---|---|
| Stage 0 | 0% | - | All tests pass |
| Stage 1 | 1% | 1 hour | Latency p99 < threshold |
| Stage 2 | 5% | 4 hours | Quality metrics stable |
| Stage 3 | 25% | 24 hours | No safety regressions |
| Stage 4 | 50% | 48 hours | User satisfaction maintained |
| Stage 5 | 100% | - | Full rollout |
Blue-Green Deployment
Shadow Deployment
Automated Rollback
Rollback Triggers
Recovery Procedures
Automated Recovery:
- Detect threshold breach
- Pause new traffic to canary
- Drain in-flight requests (graceful shutdown)
- Redirect traffic to previous version
- Notify on-call team
- Log incident for post-mortem
Evaluation Gates
Pre-Deployment Evaluation
Before any rollout begins, the new version must pass automated evaluation:
| Gate | Metric | Threshold | Action on Failure |
|---|---|---|---|
| Safety | Toxicity score | < 0.05 | Block deployment |
| Quality | Win rate vs. current | > 50% | Block deployment |
| Performance | Latency p99 | < current + 20% | Warning |
| Robustness | Adversarial test suite | 100% pass | Block deployment |
Post-Deployment Monitoring
Practice Exercises
-
Conceptual: Explain why model versioning must include the system prompt and configuration, not just the model weights. What can go wrong with weight-only versioning?
-
Mathematical: Design a canary deployment schedule for a 70B model rollout, given that latency regression can only be detected with 95% confidence after 10,000 requests.
-
Practical: Implement a blue-green deployment system for an LLM service that supports instant rollback, traffic splitting, and automated health checks.
-
Research: Compare the risk profiles of canary deployments versus shadow deployments for LLM systems. Under what conditions is each approach preferable?
What to Learn Next
-> AB Testing for LLMs Experiment design, statistical significance, and canary deployments.
-> LLM Fine-Tuning Pipelines End-to-end fine-tuning infrastructure and data management.
-> LLM Monitoring and Observability Logging, tracing, metrics, and drift detection for production systems.
-> LLM Evaluation in Production Online evaluation, user feedback loops, and quality assurance.
-> LLM Disaster Recovery Failover, backup models, and graceful degradation strategies.
-> LLM Serving Architectures vLLM, TGI, TensorRT-LLM, and serving patterns for production deployments.