ML Engineering
ML System Design — Building Production ML Systems at Scale
Master the architecture and design patterns for building robust, scalable machine learning systems in production.
- Feature Stores — Centralized feature management for consistency
- Model Serving — Real-time and batch prediction architectures
- Monitoring and Observability — Ensuring models perform well in production
"A model is only as good as the system that serves it."
📋 Prerequisites
- ● Machine Learning Fundamentals: Model training, evaluation, and deployment concepts
- ● Software Engineering: APIs, databases, containerization, CI/CD
- ● Cloud Computing: AWS/GCP/Azure basics, Kubernetes, Docker
- ● Data Engineering: ETL pipelines, data warehouses, streaming systems
- ● Python & SQL: Data manipulation, query optimization
🎯 Learning Objectives
ML System Design — Complete Guide
ML system design combines software engineering with ML to build reliable, scalable production systems.
ML System Architecture
Key Formulas Reference
Key Formulas — ML System Design
L = concurrent requests, λ = arrival rate, W = average service time
p = parallel fraction, n = processors
P = expected distribution, Q = actual distribution, PSI > 0.1 = drift
Online store: Redis/DynamoDB, Offline store: S3/BigQuery
Batch is 10-100× cheaper than real-time per prediction
Real-Time vs Batch Serving
Feature Store Architecture
Model Monitoring & Drift Detection
Python Implementation Example
from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np
import redis
import json
from typing import List
app = FastAPI()
redis_client = redis.Redis(host='localhost', port=6379, db=0)
class PredictionRequest(BaseModel):
user_id: str
item_features: List[float]
class FeatureStore:
def __init__(self, redis_client):
self.redis = redis_client
def get_features(self, entity_id: str, feature_names: List[str]) -> dict:
features = {}
for name in feature_names:
value = self.redis.hget(f"features:{entity_id}", name)
if value:
features[name] = json.loads(value)
return features
def set_features(self, entity_id: str, features: dict):
self.redis.hset(f"features:{entity_id}", mapping={
k: json.dumps(v) for k, v in features.items()
})
class ModelServer:
def __init__(self, model_path):
self.model = self.load_model(model_path)
def predict(self, features: dict) -> float:
X = self.preprocess(features)
return float(self.model.predict(X)[0])
def preprocess(self, features: dict) -> np.ndarray:
return np.array([list(features.values())])
feature_store = FeatureStore(redis_client)
model_server = ModelServer("model.pkl")
@app.post("/predict")
async def predict(request: PredictionRequest):
# 1. Get features from feature store
features = feature_store.get_features(
request.user_id,
["user_clicks", "user_purchases", "item_popularity"]
)
# 2. Get model prediction
prediction = model_server.predict(features)
# 3. Log prediction for monitoring
log_prediction(request.user_id, prediction)
return {"prediction": prediction, "features_used": list(features.keys())}
@app.post("/predict_batch")
async def predict_batch(user_ids: List[str]):
# Batch prediction for efficiency
results = []
for user_id in user_ids:
features = feature_store.get_features(user_id, [...])
prediction = model_server.predict(features)
results.append({"user_id": user_id, "prediction": prediction})
return results
# Monitoring endpoint
@app.get("/monitoring/drift")
async def check_drift():
recent_predictions = get_recent_predictions(last_hour=True)
baseline_distribution = get_baseline_distribution()
psi = calculate_psi(baseline_distribution, recent_predictions)
return {
"psi_score": psi,
"drift_detected": psi > 0.1,
"recommendation": "Retrain model" if psi > 0.25 else "Monitor"
}
Real-World Applications
🌍 Real-World Applications of ML System Design
1. Recommendation Systems (Netflix, Spotify)
Netflix serves 250M+ users with real-time recommendations. Their system uses feature stores for user/item features, real-time serving with <100ms latency, A/B testing for model comparison, and batch prediction for email campaigns. Feature engineering pipelines process billions of events daily.
2. Fraud Detection (PayPal, Stripe)
Financial systems process millions of transactions per second with real-time fraud scoring. The architecture includes streaming feature computation (transaction velocity, merchant risk), online feature stores for low-latency lookups, and model serving with fallback rules for high-confidence cases.
3. Search Ranking (Google, Bing)
Search engines use ML systems to rank billions of documents per query. The architecture includes feature stores for document/query features, real-time serving at 100K+ QPS, and continuous model updates via online learning. Latency is critical — every 100ms delay costs 1% revenue.
4. Autonomous Vehicles (Tesla, Waymo)
Self-driving cars require ML systems with edge deployment. Models run on car hardware with TFLite/ONNX, sensor data is processed in real-time, and fleet learning sends model updates to all vehicles. Monitoring tracks prediction confidence and triggers human oversight.
5. Healthcare Diagnostics
Hospital ML systems process medical images and patient data with strict privacy requirements. Architecture includes federated learning across hospitals, HIPAA-compliant feature stores, batch prediction for reports, and monitoring for model performance degradation on new patient populations.
6. Ad Tech (Google Ads, Facebook)
Ad bidding systems serve predictions at microsecond latency. Architecture includes feature stores with user/ advertiser features, real-time model serving on GPUs, A/B testing for CTR models, and monitoring for prediction quality and revenue impact.
Common Mistakes & How to Avoid Them
⚠️ Common Mistakes & How to Avoid Them
- 1Training-Serving Skew:
Using different feature computation code in training vs serving. Always use the same feature pipeline (feature store) for both. Implement point-in-time feature lookups to prevent data leakage.
- 2No Model Versioning:
Deploying models without version tracking makes rollback impossible. Use model registries (MLflow, Sagemaker) to version models, track lineage, and enable instant rollback when issues are detected.
- 3Ignoring Latency Requirements:
Designing for throughput without considering latency. Always define latency SLAs (p50, p95, p99) upfront. Use model optimization (quantization, distillation) to meet latency targets.
- 4No Monitoring or Alerting:
Deploying models without monitoring for drift or degradation. Always track: prediction distribution, feature distributions, model performance (when labels available), and system metrics (latency, error rates).
- 5Over-Engineering Early:
Building complex distributed systems when a simple API suffices. Start with the simplest architecture that meets requirements. Add complexity (feature stores, A/B testing, autoscaling) only when needed.
- 6Skipping A/B Testing:
Rolling out model changes without validation. Always A/B test new models against production baselines. Measure impact on business metrics (revenue, engagement), not just model metrics (accuracy, AUC).
Interview Questions
💬 Interview Questions — ML System Design
Q1: Design a real-time recommendation system.
Key components: (1) Feature store with user/item features (Redis for online, S3 for offline), (2) Candidate generation (ANN index for fast retrieval), (3) Ranking model (gradient boosting or neural network), (4) Real-time serving (TF Serving on Kubernetes), (5) A/B testing framework, (6) Monitoring for CTR and latency.
Q2: How do you handle training-serving skew?
Solutions: (1) Use same feature computation code for training and serving, (2) Feature store with point-in-time correct lookups, (3) Log serving features and compare with training features, (4) Schema validation at serving time, (5) Feature monitoring and alerting for distribution shifts.
Q3: When would you choose batch vs real-time serving?
Real-time when: latency requirement <100ms, user-facing applications, dynamic data. Batch when: offline processing, large volume tolerance, cost sensitivity, non-time-critical. Many systems use both: batch for bulk scoring, real-time for urgent predictions.
Q4: How do you monitor ML models in production?
Monitor: (1) Prediction distribution (drift detection), (2) Feature distributions (data drift), (3) Model performance when labels arrive (delayed), (4) System metrics (latency, throughput, error rates), (5) Business metrics (revenue, engagement). Set up alerts for anomalies and automated rollback triggers.
Q5: What is a feature store and why use one?
A feature store centralizes feature computation and storage, ensuring training-serving consistency. Benefits: (1) Same features used in training and serving, (2) Feature reuse across teams, (3) Point-in-time correctness (no data leakage), (4) Reduced engineering time, (5) Feature versioning for reproducibility.
Q6: How do you design for fault tolerance?
Strategies: (1) Model fallback (rule-based or previous model), (2) Circuit breakers for model serving, (3) Feature cache for Redis failures, (4) Graceful degradation (return average prediction), (5) Health checks and auto-restart, (6) Blue-green deployments for zero-downtime updates.
Q7: How do you estimate system capacity?
Calculate: (1) QPS = requests per second, (2) Latency budget = p99 target, (3) Throughput = QPS × average payload size, (4) Storage = data volume × retention period, (5) Compute = model complexity × QPS. Use Little's Law: concurrent_requests = QPS × avg_latency. Add 2-3× headroom for peaks.
Practice Exercise
🏋️ Practice Exercise — ML System Design Challenge
Challenge:
Design and implement a simplified ML serving system:
- Build a FastAPI model serving endpoint with <100ms latency target
- Implement a simple feature store using Redis for caching
- Add request logging and basic monitoring (prediction distribution tracking)
- Implement model versioning and A/B testing (10% traffic to new model)
- Add health checks, circuit breakers, and fallback predictions
- Deploy with Docker and write a simple Kubernetes manifest
Deliverables:
- FastAPI application with model serving endpoints
- Redis-backed feature store implementation
- Prometheus metrics for monitoring
- Docker container with health checks
- Load test results showing latency distribution
- Documentation of design decisions and trade-offs
Comparison Table
📊 ML System Design: Real-Time vs Batch Comparison
| Aspect | Real-Time Serving | Batch Prediction | Streaming |
|---|---|---|---|
| Latency | < 100ms (p99) | Minutes to hours | 100ms - 5s |
| Throughput | 100 - 100K QPS | 10M - 1B records/batch | 1K - 100K events/s |
| Cost per Prediction | High (GPU, low-latency) | Low (CPU, scheduled) | Medium (streaming infra) |
| Use Cases | Recsys, fraud, search | Reports, emails, scoring | IoT, logs, real-time analytics |
| Infrastructure | K8s, TF Serving, Triton | Spark, Airflow, dbt | Kafka, Flink, Spark Streaming |
| Model Complexity | Limited by latency budget | No latency constraints | Moderate complexity |
| Autoscaling | Required (variable QPS) | Scheduled (predictable) | Required (variable load) |
| Complexity | High | Low-Medium | High |
Key Takeaways
📌 Key Takeaways — ML System Design
- ▸ ML systems require 4 layers: data, training, serving, monitoring
- ▸ Feature stores ensure training-serving consistency (Feast, Tecton)
- ▸ Real-time serving needs sub-100ms latency (TF Serving, Triton)
- ▸ Batch prediction for offline processing at scale (Spark, Airflow)
- ▸ Model registries version and track models (MLflow)
- ▸ Monitoring detects data drift and performance degradation
- ▸ A/B testing validates model updates before full rollout
- ▸ Scalability requires Kubernetes, autoscaling, and proper infrastructure
- ▸ Training-serving skew is the #1 cause of production ML failures
- ▸ Start simple — add complexity only when requirements demand it
- ▸ Fault tolerance requires fallback strategies and circuit breakers
- ▸ Monitor business metrics, not just model metrics — revenue, engagement
- ▸ Feature versioning enables reproducible model training
- ▸ Point-in-time correctness prevents data leakage in feature stores
What to Learn Next
-> MLOps — Machine Learning Operations Complete Guide Learn about mlops — machine learning operations complete guide.
-> Model Deployment — APIs, Containers and Production ML Learn about model deployment — apis, containers and production ml.
-> Model Evaluation — Metrics, Cross-Validation and Selection Learn about model evaluation — metrics, cross-validation and selection.
-> Feature Stores — Managing ML Features at Scale Learn about feature stores — managing ml features at scale.
-> Capstone Projects — End-to-End ML Applications Learn about capstone projects — end-to-end ml applications.
-> Model Deployment — APIs, Containers and Production ML Learn about model deployment — apis, containers and production ml.
Further Reading
📚 Further Reading
- 📄 Google, "Hidden Technical Debt in Machine Learning Systems" (2015) — Classic paper on ML system complexity
- 📄 Amershi et al., "Software Engineering for ML" (2019) — ML engineering best practices
- 📄 Polyzotis et al., "Data Management Challenges in Production ML" (2017) — Feature store motivation
- 📄 Sculley et al., "Machine Learning: The High-Interest Credit Card of Technical Debt" (2014) — ML technical debt
- 📖 "Designing Machine Learning Systems" by Chip Huyen (O'Reilly, 2022) — Comprehensive ML systems book
- 📖 "Machine Learning Engineering" by Andriy Burkov (True Positive, 2020) — Practical ML engineering guide
- 🔗 Google ML Best Practices: https://developers.google.com/machine-learning/guides