ML System Design

1. End-to-End ML System Architecture

2. Feature Stores

2.1 Purpose

A feature store is a centralized platform for storing, managing, and serving ML features. It solves key challenges:

1. Consistency: Same feature computation for training and serving
2. Reuse: Share features across teams and models
3. Low latency: Serve features in milliseconds for online inference
4. Point-in-time correctness: Prevent data leakage in training

2.2 Feature Store Architecture

where:

• : Low-latency store (Redis, DynamoDB) for serving
• : High-throughput store (S3, Hive) for training
• : Feature registry with metadata and lineage

2.3 Online vs Offline Serving

2.4 Point-in-Time Correctness

For training, features must be computed using only data available at the time of the label:

This prevents data leakage where future information leaks into training features.

3. Model Serving

3.1 Latency Budget Analysis

For a real-time prediction service:

3.2 Serving Patterns

Sync serving: Wait for prediction before returning:

Async serving: Return immediately, process in background:

Batch serving: Accumulate requests, process in batches:

3.3 Model Optimization for Serving

Quantization: Reduce precision from FP32 to INT8/INT4:

Knowledge distillation: Train smaller student model:

Pruning: Remove redundant parameters:

ONNX/TensorRT optimization: Graph optimization, kernel fusion, layer fusion.

4. A/B Testing

4.1 Statistical Framework

Null hypothesis : No difference between model variants A and B.

Test statistic for continuous metric (e.g., revenue per user):

where is the sample mean, is the sample variance, and is the sample size.

4.2 Statistical Power

The power of a test is the probability of correctly rejecting when the alternative is true:

where:

• : minimum detectable effect
• : sample size per group
• : standard deviation
• : significance level (typically 0.05)

4.3 Sample Size Calculation

For , (80% power):

4.4 Multiple Testing Correction

When testing multiple metrics, apply Bonferroni correction:

where is the number of metrics tested.

4.5 Sequential Testing

For experiments that run continuously, use sequential testing to stop early:

Always-Valid P-values:

This allows monitoring results without inflating Type I error.

5. Monitoring and Observability

5.1 Data Drift Detection

Covariate shift: Distribution of input features changes:

Population Stability Index (PSI):

where and are the proportions in bin for training and serving distributions.

Interpretation:

• PSI < 0.1: No significant drift
• 0.1 ≤ PSI < 0.25: Moderate drift
• PSI ≥ 0.25: Significant drift

5.2 Model Performance Monitoring

Prediction drift: Distribution of model outputs changes:

Concept drift: Relationship between inputs and outputs changes:

5.3 Alerting Rules

where is typically 3 (3-sigma rule) or based on domain knowledge.

5.4 Monitoring Dashboard Metrics

6. Feature Pipelines

6.1 Batch Feature Computation

Computed periodically (hourly/daily) using Spark, Flink, or similar.

6.2 Streaming Feature Computation

Computed on each event using streaming frameworks (Kafka Streams, Flink).

6.3 Feature Transformation Patterns

Aggregation: Sum, count, average over time window:

Embedding: Map categorical to dense vector:

Interaction: Cross features:

Temporal: Time-based features:

7. Model Registry

7.1 Model Versioning

Each model version is tracked with:

• Artifacts: Model weights, config, tokenizer
• Metadata: Training data, hyperparameters, metrics
• Lineage: Parent model, training pipeline
• Tags: Production, staging, development

7.2 Promotion Workflow

Each transition requires:

1. Passing automated tests
2. Performance benchmarks
3. Human review (for critical models)
4. Approval from model owners

8. ML System Patterns

8.1 Online Prediction

Latency-sensitive, typically < 100ms.

8.2 Batch Prediction

Throughput-focused, processes millions of examples.

8.3 Near-Real-Time

Combines streaming features with batch model.

9. Implementation Example

from feast import FeatureStore
from fastapi import FastAPI
import redis
import numpy as np

class ModelServingService:
    def __init__(self):
        self.feature_store = FeatureStore(repo_path="feature_repo")
        self.model = load_model("model_v2.pkl")
        self.redis = redis.Redis(host="localhost", port=6379)
        self.app = FastAPI()

    async def predict(self, user_id: str, item_id: str):
        # 1. Fetch features (online)
        features = self.feature_store.get_online_features(
            features=[
                "user_features:age",
                "user_features:gender",
                "item_features:category",
                "item_features:price",
            ],
            entity_rows=[{"user_id": user_id, "item_id": item_id}]
        )

        # 2. Check cache
        cache_key = f"pred:{user_id}:{item_id}"
        cached = self.redis.get(cache_key)
        if cached:
            return float(cached)

        # 3. Model inference
        X = features.to_dict()
        prediction = self.model.predict_proba(X)[:, 1]

        # 4. Cache result
        self.redis.setex(cache_key, 300, float(prediction))

        return float(prediction)

    def log_prediction(self, user_id, item_id, prediction, label=None):
        """Log for monitoring and retraining."""
        log_entry = {
            "timestamp": time.time(),
            "user_id": user_id,
            "item_id": item_id,
            "prediction": prediction,
            "label": label
        }
        self.logger.info(log_entry)

10. Cost Analysis

10.1 Total Cost of Ownership

10.2 Cost Optimization Strategies

1. Auto-scaling: Match compute to demand
2. Spot instances: Use preemptible instances for training
3. Model optimization: Smaller models = lower serving cost
4. Caching: Reduce redundant inference
5. Batch processing: Amortize fixed costs

11. Best Practices

1. Start simple: Begin with basic infrastructure, add complexity as needed
2. Automate everything: CI/CD for models, automated testing
3. Monitor comprehensively: Track data, model, and system metrics
4. Version everything: Data, features, models, configurations
5. Plan for failure: Graceful degradation, fallback models
6. Document decisions: Architecture decision records (ADRs)

References

1. Huyen. "Designing Machine Learning Systems." O'Reilly, 2022.
2. Lewis et al. (2021). "Feature Stores for Machine Learning." ACM Computing Surveys.
3. Kumar et al. (2019). "Challenges of Online A/B Testing." arXiv.
4. Polyzotis et al. (2017). "Data Management Challenges in Production Machine Learning." SIGMOD.
5. Sculley et al. (2015). "Hidden Technical Debt in Machine Learning Systems." NeurIPS.