🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
💼 Servicesℹ️ About✉️ ContactView Pricing Plansfrom $10

ML System Design — Architecture and Production Patterns

Expert TopicsSystem Design🟢 Free Lesson

Advertisement

ML Engineering

ML System Design — Building Production ML Systems at Scale

Master the architecture and design patterns for building robust, scalable machine learning systems in production.

  • Feature Stores — Centralized feature management for consistency
  • Model Serving — Real-time and batch prediction architectures
  • Monitoring and Observability — Ensuring models perform well in production

"A model is only as good as the system that serves it."

📋 Prerequisites

  • Machine Learning Fundamentals: Model training, evaluation, and deployment concepts
  • Software Engineering: APIs, databases, containerization, CI/CD
  • Cloud Computing: AWS/GCP/Azure basics, Kubernetes, Docker
  • Data Engineering: ETL pipelines, data warehouses, streaming systems
  • Python & SQL: Data manipulation, query optimization

🎯 Learning Objectives

Design end-to-end ML system architectures
Implement real-time and batch prediction pipelines
Build and manage feature stores for consistency
Set up model monitoring and drift detection
Apply A/B testing and canary deployments
Evaluate trade-offs between latency, throughput, and cost

ML System Design — Complete Guide

ML system design combines software engineering with ML to build reliable, scalable production systems.


ML System Architecture

Four-Layer ML System ArchitectureDATA LAYERData CollectionStreams, APIs, ETLData Lake / WarehouseS3, BigQuery, SnowflakeFeature StoreFeast, TectonData QualityGreat ExpectationsTRAINING LAYERExperiment TrackingMLflow, W and BModel TrainingDistributed GPUModel EvaluationMetrics, A/BModel RegistryVersioning, lineageSERVING LAYERReal-time API< 100ms, TF ServingBatch PredictionSpark, scheduledEdge DeploymentTFLite, ONNXA/B TestingTraffic splittingMONITORINGData DriftModel PerformanceLatency/ThroughputAlerting and RollbackFeedback Loop

Key Formulas Reference

Key Formulas — ML System Design

Little's Law (Throughput):

L = concurrent requests, λ = arrival rate, W = average service time

Amdahl's Law (Speedup):

p = parallel fraction, n = processors

Data Drift Detection (PSI):

P = expected distribution, Q = actual distribution, PSI > 0.1 = drift

Feature Store Latency Target:

Online store: Redis/DynamoDB, Offline store: S3/BigQuery

Model Serving Cost:

Batch is 10-100× cheaper than real-time per prediction


Real-Time vs Batch Serving

Real-Time vs Batch PredictionReal-Time InferenceLatency: < 100ms (p99)Pattern: Request → ResponseUse: Recsys, fraud, search rankingStack: TF Serving, Triton, BentoMLInfra: Kubernetes, autoscalingCost: High (GPU/low-latency infra)QPS: 100 - 100,000+Batch PredictionLatency: minutes to hoursPattern: Schedule → Process → StoreUse: Reports, emails, scoringStack: Spark, Airflow, dbtInfra: Data lake, warehouseCost: Lower (CPU, scheduled)Records: 10M - 1B per batch

Feature Store Architecture


Model Monitoring & Drift Detection


Python Implementation Example

from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np
import redis
import json
from typing import List

app = FastAPI()
redis_client = redis.Redis(host='localhost', port=6379, db=0)

class PredictionRequest(BaseModel):
    user_id: str
    item_features: List[float]

class FeatureStore:
    def __init__(self, redis_client):
        self.redis = redis_client
    
    def get_features(self, entity_id: str, feature_names: List[str]) -> dict:
        features = {}
        for name in feature_names:
            value = self.redis.hget(f"features:{entity_id}", name)
            if value:
                features[name] = json.loads(value)
        return features
    
    def set_features(self, entity_id: str, features: dict):
        self.redis.hset(f"features:{entity_id}", mapping={
            k: json.dumps(v) for k, v in features.items()
        })

class ModelServer:
    def __init__(self, model_path):
        self.model = self.load_model(model_path)
    
    def predict(self, features: dict) -> float:
        X = self.preprocess(features)
        return float(self.model.predict(X)[0])
    
    def preprocess(self, features: dict) -> np.ndarray:
        return np.array([list(features.values())])

feature_store = FeatureStore(redis_client)
model_server = ModelServer("model.pkl")

@app.post("/predict")
async def predict(request: PredictionRequest):
    # 1. Get features from feature store
    features = feature_store.get_features(
        request.user_id,
        ["user_clicks", "user_purchases", "item_popularity"]
    )
    
    # 2. Get model prediction
    prediction = model_server.predict(features)
    
    # 3. Log prediction for monitoring
    log_prediction(request.user_id, prediction)
    
    return {"prediction": prediction, "features_used": list(features.keys())}

@app.post("/predict_batch")
async def predict_batch(user_ids: List[str]):
    # Batch prediction for efficiency
    results = []
    for user_id in user_ids:
        features = feature_store.get_features(user_id, [...])
        prediction = model_server.predict(features)
        results.append({"user_id": user_id, "prediction": prediction})
    return results

# Monitoring endpoint
@app.get("/monitoring/drift")
async def check_drift():
    recent_predictions = get_recent_predictions(last_hour=True)
    baseline_distribution = get_baseline_distribution()
    
    psi = calculate_psi(baseline_distribution, recent_predictions)
    
    return {
        "psi_score": psi,
        "drift_detected": psi > 0.1,
        "recommendation": "Retrain model" if psi > 0.25 else "Monitor"
    }

Real-World Applications

🌍 Real-World Applications of ML System Design

1. Recommendation Systems (Netflix, Spotify)

Netflix serves 250M+ users with real-time recommendations. Their system uses feature stores for user/item features, real-time serving with <100ms latency, A/B testing for model comparison, and batch prediction for email campaigns. Feature engineering pipelines process billions of events daily.

2. Fraud Detection (PayPal, Stripe)

Financial systems process millions of transactions per second with real-time fraud scoring. The architecture includes streaming feature computation (transaction velocity, merchant risk), online feature stores for low-latency lookups, and model serving with fallback rules for high-confidence cases.

3. Search Ranking (Google, Bing)

Search engines use ML systems to rank billions of documents per query. The architecture includes feature stores for document/query features, real-time serving at 100K+ QPS, and continuous model updates via online learning. Latency is critical — every 100ms delay costs 1% revenue.

4. Autonomous Vehicles (Tesla, Waymo)

Self-driving cars require ML systems with edge deployment. Models run on car hardware with TFLite/ONNX, sensor data is processed in real-time, and fleet learning sends model updates to all vehicles. Monitoring tracks prediction confidence and triggers human oversight.

5. Healthcare Diagnostics

Hospital ML systems process medical images and patient data with strict privacy requirements. Architecture includes federated learning across hospitals, HIPAA-compliant feature stores, batch prediction for reports, and monitoring for model performance degradation on new patient populations.

6. Ad Tech (Google Ads, Facebook)

Ad bidding systems serve predictions at microsecond latency. Architecture includes feature stores with user/ advertiser features, real-time model serving on GPUs, A/B testing for CTR models, and monitoring for prediction quality and revenue impact.


Common Mistakes & How to Avoid Them

⚠️ Common Mistakes & How to Avoid Them

  • 1
    Training-Serving Skew:

    Using different feature computation code in training vs serving. Always use the same feature pipeline (feature store) for both. Implement point-in-time feature lookups to prevent data leakage.

  • 2
    No Model Versioning:

    Deploying models without version tracking makes rollback impossible. Use model registries (MLflow, Sagemaker) to version models, track lineage, and enable instant rollback when issues are detected.

  • 3
    Ignoring Latency Requirements:

    Designing for throughput without considering latency. Always define latency SLAs (p50, p95, p99) upfront. Use model optimization (quantization, distillation) to meet latency targets.

  • 4
    No Monitoring or Alerting:

    Deploying models without monitoring for drift or degradation. Always track: prediction distribution, feature distributions, model performance (when labels available), and system metrics (latency, error rates).

  • 5
    Over-Engineering Early:

    Building complex distributed systems when a simple API suffices. Start with the simplest architecture that meets requirements. Add complexity (feature stores, A/B testing, autoscaling) only when needed.

  • 6
    Skipping A/B Testing:

    Rolling out model changes without validation. Always A/B test new models against production baselines. Measure impact on business metrics (revenue, engagement), not just model metrics (accuracy, AUC).


Interview Questions

💬 Interview Questions — ML System Design

Q1: Design a real-time recommendation system.

Key components: (1) Feature store with user/item features (Redis for online, S3 for offline), (2) Candidate generation (ANN index for fast retrieval), (3) Ranking model (gradient boosting or neural network), (4) Real-time serving (TF Serving on Kubernetes), (5) A/B testing framework, (6) Monitoring for CTR and latency.

Q2: How do you handle training-serving skew?

Solutions: (1) Use same feature computation code for training and serving, (2) Feature store with point-in-time correct lookups, (3) Log serving features and compare with training features, (4) Schema validation at serving time, (5) Feature monitoring and alerting for distribution shifts.

Q3: When would you choose batch vs real-time serving?

Real-time when: latency requirement <100ms, user-facing applications, dynamic data. Batch when: offline processing, large volume tolerance, cost sensitivity, non-time-critical. Many systems use both: batch for bulk scoring, real-time for urgent predictions.

Q4: How do you monitor ML models in production?

Monitor: (1) Prediction distribution (drift detection), (2) Feature distributions (data drift), (3) Model performance when labels arrive (delayed), (4) System metrics (latency, throughput, error rates), (5) Business metrics (revenue, engagement). Set up alerts for anomalies and automated rollback triggers.

Q5: What is a feature store and why use one?

A feature store centralizes feature computation and storage, ensuring training-serving consistency. Benefits: (1) Same features used in training and serving, (2) Feature reuse across teams, (3) Point-in-time correctness (no data leakage), (4) Reduced engineering time, (5) Feature versioning for reproducibility.

Q6: How do you design for fault tolerance?

Strategies: (1) Model fallback (rule-based or previous model), (2) Circuit breakers for model serving, (3) Feature cache for Redis failures, (4) Graceful degradation (return average prediction), (5) Health checks and auto-restart, (6) Blue-green deployments for zero-downtime updates.

Q7: How do you estimate system capacity?

Calculate: (1) QPS = requests per second, (2) Latency budget = p99 target, (3) Throughput = QPS × average payload size, (4) Storage = data volume × retention period, (5) Compute = model complexity × QPS. Use Little's Law: concurrent_requests = QPS × avg_latency. Add 2-3× headroom for peaks.


Practice Exercise

🏋️ Practice Exercise — ML System Design Challenge

Challenge:

Design and implement a simplified ML serving system:

  1. Build a FastAPI model serving endpoint with <100ms latency target
  2. Implement a simple feature store using Redis for caching
  3. Add request logging and basic monitoring (prediction distribution tracking)
  4. Implement model versioning and A/B testing (10% traffic to new model)
  5. Add health checks, circuit breakers, and fallback predictions
  6. Deploy with Docker and write a simple Kubernetes manifest

Deliverables:

  • FastAPI application with model serving endpoints
  • Redis-backed feature store implementation
  • Prometheus metrics for monitoring
  • Docker container with health checks
  • Load test results showing latency distribution
  • Documentation of design decisions and trade-offs

Comparison Table

📊 ML System Design: Real-Time vs Batch Comparison

AspectReal-Time ServingBatch PredictionStreaming
Latency< 100ms (p99)Minutes to hours100ms - 5s
Throughput100 - 100K QPS10M - 1B records/batch1K - 100K events/s
Cost per PredictionHigh (GPU, low-latency)Low (CPU, scheduled)Medium (streaming infra)
Use CasesRecsys, fraud, searchReports, emails, scoringIoT, logs, real-time analytics
InfrastructureK8s, TF Serving, TritonSpark, Airflow, dbtKafka, Flink, Spark Streaming
Model ComplexityLimited by latency budgetNo latency constraintsModerate complexity
AutoscalingRequired (variable QPS)Scheduled (predictable)Required (variable load)
ComplexityHighLow-MediumHigh

Key Takeaways

📌 Key Takeaways — ML System Design

  • ML systems require 4 layers: data, training, serving, monitoring
  • Feature stores ensure training-serving consistency (Feast, Tecton)
  • Real-time serving needs sub-100ms latency (TF Serving, Triton)
  • Batch prediction for offline processing at scale (Spark, Airflow)
  • Model registries version and track models (MLflow)
  • Monitoring detects data drift and performance degradation
  • A/B testing validates model updates before full rollout
  • Scalability requires Kubernetes, autoscaling, and proper infrastructure
  • Training-serving skew is the #1 cause of production ML failures
  • Start simple — add complexity only when requirements demand it
  • Fault tolerance requires fallback strategies and circuit breakers
  • Monitor business metrics, not just model metrics — revenue, engagement
  • Feature versioning enables reproducible model training
  • Point-in-time correctness prevents data leakage in feature stores

What to Learn Next

-> MLOps — Machine Learning Operations Complete Guide Learn about mlops — machine learning operations complete guide.

-> Model Deployment — APIs, Containers and Production ML Learn about model deployment — apis, containers and production ml.

-> Model Evaluation — Metrics, Cross-Validation and Selection Learn about model evaluation — metrics, cross-validation and selection.

-> Feature Stores — Managing ML Features at Scale Learn about feature stores — managing ml features at scale.

-> Capstone Projects — End-to-End ML Applications Learn about capstone projects — end-to-end ml applications.

-> Model Deployment — APIs, Containers and Production ML Learn about model deployment — apis, containers and production ml.


Further Reading

📚 Further Reading

  • 📄 Google, "Hidden Technical Debt in Machine Learning Systems" (2015) — Classic paper on ML system complexity
  • 📄 Amershi et al., "Software Engineering for ML" (2019) — ML engineering best practices
  • 📄 Polyzotis et al., "Data Management Challenges in Production ML" (2017) — Feature store motivation
  • 📄 Sculley et al., "Machine Learning: The High-Interest Credit Card of Technical Debt" (2014) — ML technical debt
  • 📖 "Designing Machine Learning Systems" by Chip Huyen (O'Reilly, 2022) — Comprehensive ML systems book
  • 📖 "Machine Learning Engineering" by Andriy Burkov (True Positive, 2020) — Practical ML engineering guide
  • 🔗 Google ML Best Practices: https://developers.google.com/machine-learning/guides

Need Expert Machine Learning Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement