Interview Question

"Explain the challenges of deploying ML models in production. How do you set up A/B testing for model comparison? What is model drift and how do you detect it?"

Difficulty: Hard | Frequently asked at Google, Netflix, Amazon

Theoretical Foundation

Model Deployment Challenges

1. Scalability: Serving millions of requests per second
2. Latency: Real-time predictions in milliseconds
3. Reliability: 99.99% uptime requirements
4. Monitoring: Detecting model degradation
5. Versioning: Managing multiple model versions
6. Security: Protecting against adversarial attacks

Model Serving Architectures

Batch Serving

• Pre-compute predictions offline
• Store in database/cache
• Serve via lookup
• Use case: Recommendations, daily reports

Real-time Serving

• Compute predictions on-demand
• Requires low-latency inference
• Use case: Fraud detection, search ranking

Streaming Serving

• Process data streams in real-time
• Update predictions incrementally
• Use case: IoT, real-time monitoring

Model Serving Optimization

1. Model Compression: Pruning, quantization, knowledge distillation
2. Caching: Cache frequent predictions
3. Batching: Process multiple requests together
4. Hardware Acceleration: GPU, TPU, FPGA
5. Edge Deployment: Deploy to edge devices

A/B Testing for ML

Setup:

1. Split traffic between model variants
2. Random assignment ensures unbiased comparison
3. Statistical significance testing

Key Metrics:

• Online metrics: CTR, conversion rate, revenue
• Model metrics: Accuracy, latency, throughput
• Business metrics: ROI, customer satisfaction

Statistical Testing:

• t-test: Compare means of two groups
• Chi-squared test: Compare proportions
• Bayesian testing: Probabilistic comparison

Model Drift

Concept Drift

The relationship between features and target changes over time.

Data Drift

The distribution of input features changes over time.

Detection Methods

1. Statistical Tests:

   • KS test for distribution changes
   • PSI (Population Stability Index)
   • Chi-squared test for categorical features

2. Performance Monitoring:

   • Track prediction accuracy over time
   • Monitor error rates

3. Data Quality Checks:

   • Missing value rates
   • Feature distribution shifts

PSI (Population Stability Index)

where $p_i$ is the proportion in bin $i$ for current data, $q_i$ for reference data.

Interpretation:

• PSI < 0.1: No significant change
• 0.1 < PSI < 0.25: Moderate change
• PSI > 0.25: Significant change (investigate)

MLOps Pipeline

1. Data Pipeline: Ingestion, validation, preprocessing
2. Training Pipeline: Model training, evaluation, versioning
3. Deployment Pipeline: Model serving, A/B testing
4. Monitoring Pipeline: Drift detection, alerting

Code Implementation

Real-World Applications

Google: Search Ranking Deployment

• Model Serving: TensorFlow Serving at scale
• A/B Testing: Continuous model improvement
• Drift Detection: Monitoring search quality

Netflix: Recommendation Deployment

• Real-time Serving: Sub-100ms latency requirements
• Champion-Challenger: Always testing new models
• Personalization: Per-user model selection

Common Follow-Up Questions

Q1: How do you handle model versioning in production? Use a model registry (MLflow, SageMaker) to track versions, metadata, and lineage. Always keep previous versions for rollback.

Q2: What is shadow deployment? Run the new model alongside the old one, but only use the old model's predictions. Compare performance without affecting users.

Q3: How do you handle cold start problems? Use default models, content-based filtering, or popularity-based recommendations until enough user data is collected.

Q4: What is the difference between online and batch learning? Batch learning retrains on historical data periodically. Online learning updates incrementally with new data.

Related Topics

• MLOps Pipeline Design
• Model Monitoring
• A/B Testing Statistical Design
• Production ML Systems