Interview Question

"Design a recommendation system like Netflix or YouTube that can serve personalized content suggestions to hundreds of millions of users from a catalog of millions of items, with sub-100ms latency requirements."

Difficulty: Hard | Frequently asked at Netflix, YouTube/Google, Spotify, Amazon, Meta

1. Requirements Gathering

Functional Requirements

Before diving into architecture, we must clarify what the system needs to do:

1. Personalized Recommendations: Each user sees a unique homepage tailored to their preferences, not just popular content
2. Multiple Recommendation Surfaces: Different sections on the homepage (e.g., "Trending Now", "Because You Watched X", "Top Picks for You")
3. Real-time Adaptation: Recommendations update as users watch, rate, or interact with content
4. Cold Start Handling: System must work for new users and new content items
5. Search Integration: Related recommendations when viewing specific content detail pages
6. Cross-device Consistency: Same recommendations across mobile, TV, web

Non-Functional Requirements

1. Latency: < 100ms for feed generation, < 200ms for full page load
2. Throughput: 500M+ requests/day, peak QPS of 50,000+
3. Availability: 99.99% uptime (Netflix-level)
4. Freshness: New content should appear in recommendations within hours, user preference updates within minutes
5. Scalability: Linear horizontal scaling as users/content grow
6. Storage: Efficient storage for billions of interaction events and millions of item embeddings

2. High-Level Architecture Overview

The recommendation system follows a classic three-stage architecture: Candidate Generation → Ranking → Post-processing.

┌─────────────────────────────────────────────────────────────────────────────┐
│                           CLIENT LAYER                                      │
│  Mobile App │ Smart TV │ Web Browser │ Set-Top Box │ Gaming Console         │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                         API GATEWAY / CDN                                    │
│  Rate Limiting │ Authentication │ Load Balancing │ Response Caching          │
└─────────────────────────────────────────────────────────────────────────────┘
                                    │
                    ┌───────────────┼───────────────┐
                    ▼               ▼               ▼
┌────────────────────────┐ ┌───────────────┐ ┌──────────────────────┐
│  RECOMMENDATION SERVICE │ │ USER SERVICE  │ │ CONTENT METADATA     │
│  (Core ML Pipeline)     │ │ (Profile Mgmt)│ │ SERVICE              │
└────────────────────────┘ └───────────────┘ └──────────────────────┘
                    │
        ┌───────────┼───────────┐
        ▼           ▼           ▼
┌──────────────┐ ┌──────────┐ ┌──────────────┐
│  CANDIDATE   │ │  RANKING │ │  POST-       │
│  GENERATION  │ │  MODEL   │ │  PROCESSING  │
│  (1000s)     │ │  (100s)  │ │  (50-100)    │
└──────────────┘ └──────────┘ └──────────────┘
        │           │           │
        └───────────┼───────────┘
                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        FEATURE STORE / DATA LAYER                           │
│  Redis Cluster │ Kafka Streams │ HDFS/S3 │ Feature Pipeline │ Model Store  │
└─────────────────────────────────────────────────────────────────────────────┘

3. Data Pipeline Design

3.1 Data Sources and Collection

User Interaction Data (implicit feedback):

• Watch history (what, when, duration, completion rate)
• Search queries
• Browse patterns (click, hover, scroll depth)
• Add to list / remove from list
• Like/dislike ratings
• Skip patterns (what was skipped and when)

User Profile Data:

• Demographics (age, location, language preferences)
• Subscription tier
• Device preferences
• Explicit preferences (genre selections during onboarding)

Content Metadata:

• Title, description, cast, director
• Genre tags, content ratings
• Duration, release date
• Visual embeddings (thumbnail analysis)
• Audio features (tempo, mood)
• Text embeddings (synopsis analysis)

# Example: Event schema for user interactions
{
    "event_id": "uuid-v4",
    "user_id": "user_12345",
    "event_type": "watch_complete",  # watch_start, watch_complete, skip, like, search
    "content_id": "movie_67890",
    "timestamp": "2026-06-29T14:30:00Z",
    "duration_watched": 3600,  # seconds
    "total_duration": 5400,  # content total duration
    "device_type": "smart_tv",
    "context": {
        "time_of_day": "evening",
        "day_of_week": "saturday",
        "location_country": "US"
    }
}

3.2 Real-time Data Processing

User Actions → Kafka → Stream Processing (Flink/Spark Streaming) → Feature Store
                                                      │
                                                      ▼
                                               Aggregation Jobs
                                               (click rates, watch time)
                                                      │
                                                      ▼
                                               Batch Storage
                                               (HDFS/S3 → Data Lake)

Stream Processing Components:

1. Event Ingestion: Apache Kafka as the central event bus

   • Topic partitioning by user_id for ordering guarantees
   • 7-day retention for replay capability
   • Schema registry for evolution management

2. Real-time Aggregation: Apache Flink for windowed aggregations

   • Sliding windows (last 1 hour, 24 hours, 7 days)
   • Real-time feature computation (user's recent watch patterns)
   • Anomaly detection for bot/spam filtering

3. Feature Store: Dual-layer architecture

   • Online Store (Redis/DynamoDB): Low-latency feature serving (< 5ms)
   • Offline Store (S3/HDFS): Historical features for training

3.3 Batch Data Processing

# Example: Daily batch job for computing user embeddings
from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler
import tensorflow as tf

spark = SparkSession.builder \
    .appName("UserEmbeddingJob") \
    .config("spark.executor.memory", "16g") \
    .getOrCreate()

# Read interaction data from last 90 days
interactions = spark.read.parquet("s3://data-lake/interactions/")

# Compute user interaction matrix
user_item_matrix = interactions.groupBy("user_id", "content_id") \
    .agg(
        F.sum("watch_duration").alias("total_watch"),
        F.count("events").alias("interaction_count"),
        F.collect_list("event_type").alias("event_sequence")
    )

# Generate user embeddings using collaborative filtering
# or train neural network embeddings
user_embeddings = train_user_embeddings(user_item_matrix)

# Write to feature store
user_embeddings.write \
    .mode("overwrite") \
    .parquet("s3://features/user-embeddings/")

4. Model Selection and Training Approach

4.1 Candidate Generation Models

The goal is to efficiently reduce millions of items to thousands of candidates.

Approach 1: Collaborative Filtering (Matrix Factorization)

import numpy as np
from scipy.sparse.linalg import svds

class CollaborativeFiltering:
    def __init__(self, n_factors=128):
        self.n_factors = n_factors
        
    def fit(self, user_item_matrix):
        """
        user_item_matrix: sparse matrix of shape (n_users, n_items)
        Values are implicit feedback (watch time, interaction count)
        """
        # Normalize by user activity
        user_means = np.array(user_item_matrix.mean(axis=1)).flatten()
        
        # SVD decomposition
        U, sigma, Vt = svds(
            user_item_matrix - user_means.reshape(-1, 1),
            k=self.n_factors
        )
        
        self.user_factors = U * sigma  # (n_users, n_factors)
        self.item_factors = Vt.T        # (n_items, n_factors)
        self.user_means = user_means
        
    def predict(self, user_id, item_ids):
        scores = np.dot(
            self.user_factors[user_id],
            self.item_factors[item_ids].T
        ) + self.user_means[user_id]
        return scores

Approach 2: Two-Tower Neural Retrieval (YouTube-style)

import tensorflow as tf
import tensorflow_recommenders as tfrs

class TwoTowerModel(tfrs.Model):
    def __init__(self, embedding_dim=128):
        super().__init__()
        
        # User tower
        self.user_model = tf.keras.Sequential([
            tf.keras.layers.Embedding(num_users, embedding_dim),
            tf.keras.layers.Dense(256, activation='relu'),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.Dense(128, activation='relu'),
            tf.keras.layers.Dense(embedding_dim)
        ])
        
        # Item tower
        self.item_model = tf.keras.Sequential([
            tf.keras.layers.Embedding(num_items, embedding_dim),
            tf.keras.layers.Dense(256, activation='relu'),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.Dense(128, activation='relu'),
            tf.keras.layers.Dense(embedding_dim)
        ])
        
        # Retrieval task
        self.task = tfrs.tasks.Retrieval(
            metrics=tfrs.metrics.FactorizedTopK(
                candidates=item_dataset.batch(128).map(self.item_model)
            )
        )
    
    def compute_loss(self, features, training=False):
        user_embeddings = self.user_model(features["user_id"])
        item_embeddings = self.item_model(features["item_id"])
        return self.task(user_embeddings, item_embeddings)

Approach 3: Approximate Nearest Neighbor (ANN) Index

For serving, we use FAISS or ScaNN for efficient similarity search:

import faiss
import numpy as np

class ANNIndex:
    def __init__(self, embedding_dim=128, n_lists=1000):
        # Build IVF index for fast approximate search
        quantizer = faiss.IndexFlatIP(embedding_dim)
        self.index = faiss.IndexIVFFlat(
            quantizer, embedding_dim, n_lists,
            faiss.METRIC_INNER_PRODUCT
        )
        
    def build_index(self, item_embeddings):
        """Build index from pre-computed item embeddings"""
        faiss.normalize_L2(item_embeddings)  # Normalize for cosine similarity
        self.index.train(item_embeddings)
        self.index.add(item_embeddings)
        self.index.nprobe = 100  # Number of lists to search
        
    def search(self, query_embedding, k=1000):
        """Find k nearest items for a user query"""
        faiss.normalize_L2(query_embedding)
        distances, indices = self.index.search(query_embedding, k)
        return indices, distances

4.2 Ranking Models

Once we have candidates, we need precise ranking. The ranking model is more complex but operates on fewer items.

Deep Ranking Model Architecture:

class RankingModel(tf.keras.Model):
    def __init__(self):
        super().__init__()
        
        # Feature embedding layers
        self.user_embedding = tf.keras.layers.Embedding(1000000, 64)
        self.item_embedding = tf.keras.layers.Embedding(500000, 64)
        self.genre_embedding = tf.keras.layers.Embedding(100, 16)
        self.context_embedding = tf.keras.layers.Embedding(1000, 16)
        
        # Cross feature interaction network (DCN-style)
        self.cross_network = CrossNetwork(num_layers=3, projection_dim=64)
        
        # Deep network for high-order interactions
        self.deep_network = tf.keras.Sequential([
            tf.keras.layers.Dense(512, activation='relu'),
            tf.keras.layers.BatchNormalization(),
            tf.keras.layers.Dropout(0.3),
            tf.keras.layers.Dense(256, activation='relu'),
            tf.keras.layers.BatchNormalization(),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.Dense(128, activation='relu')
        ])
        
        # Output layer
        self.output_layer = tf.keras.layers.Dense(1, activation='sigmoid')
        
    def call(self, inputs):
        # Embed all features
        user_emb = self.user_embedding(inputs['user_id'])
        item_emb = self.item_embedding(inputs['item_id'])
        genre_emb = self.genre_embedding(inputs['genre_id'])
        context_emb = self.context_embedding(inputs['context_id'])
        
        # Concatenate all features
        features = tf.concat([
            user_emb, item_emb, genre_emb, context_emb,
            inputs['user_history_length'].reshape(-1, 1),
            inputs['time_of_day'].reshape(-1, 1),
            inputs['day_of_week'].reshape(-1, 1)
        ], axis=1)
        
        # Apply cross and deep networks
        cross_output = self.cross_network(features)
        deep_output = self.deep_network(features)
        
        # Combine and predict
        combined = tf.concat([cross_output, deep_output], axis=1)
        return self.output_layer(combined)

4.3 Training Strategy

# Training pipeline configuration
training_config = {
    'candidate_generation': {
        'model': 'TwoTower',
        'embedding_dim': 128,
        'batch_size': 4096,
        'learning_rate': 0.001,
        'epochs': 10,
        'negative_sampling': 'in-batch',
        'loss': 'softmax_cross_entropy'
    },
    'ranking': {
        'model': 'DCN_v2',
        'batch_size': 1024,
        'learning_rate': 0.0001,
        'epochs': 20,
        'loss': 'binary_crossentropy',
        'features': ['user', 'item', 'context', 'cross_features']
    }
}

# Multi-task learning for different objectives
class MultiTaskRankingModel(tf.keras.Model):
    def __init__(self):
        super().__init__()
        self.shared_trunk = SharedTrunk()
        
        # Multiple prediction heads
        self.watch_prob_head = WatchProbabilityHead()
        self.watch_time_head = WatchTimeHead()
        self.rating_head = RatingHead()
        
    def call(self, inputs):
        shared_features = self.shared_trunk(inputs)
        
        return {
            'watch_probability': self.watch_prob_head(shared_features),
            'expected_watch_time': self.watch_time_head(shared_features),
            'expected_rating': self.rating_head(shared_features)
        }

5. Serving Architecture

5.1 Real-time Serving Pipeline

User Request → Load Balancer → Recommendation Service
                                       │
                                       ▼
                              ┌─────────────────┐
                              │ Feature Retrieval │
                              │ (Redis < 5ms)     │
                              └─────────────────┘
                                       │
                                       ▼
                              ┌─────────────────┐
                              │ Candidate Gen    │
                              │ (ANN Search)     │
                              │ (~20ms)          │
                              └─────────────────┘
                                       │
                                       ▼
                              ┌─────────────────┐
                              │ Ranking Model    │
                              │ (GPU Inference)  │
                              │ (~50ms)          │
                              └─────────────────┘
                                       │
                                       ▼
                              ┌─────────────────┐
                              │ Post-processing  │
                              │ (Business Rules) │
                              │ (~5ms)           │
                              └─────────────────┘
                                       │
                                       ▼
                              Response (< 100ms)

5.2 Model Serving Options

Option A: TensorFlow Serving / TorchServe

# TensorFlow Serving configuration
{
    "model_config": {
        "model_platforms": [{
            "type": "tensorflow",
            "model_config_list": {
                "config": [{
                    "name": "ranking_model",
                    "base_path": "s3://models/ranking/v2",
                    "model_platform": "tensorflow",
                    "model_version_policy": {
                        "specific": {
                            "versions": [1, 2, 3]
                        }
                    }
                }]
            }
        }]
    }
}

Option B: Triton Inference Server (Multi-framework)

# Triton model repository structure
models/
├── candidate_generation/
│   ├── config.pbtxt
│   └── 1/
│       └── model.savedmodel/
└── ranking/
    ├── config.pbtxt
    └── 1/
        └── model.pt

Option C: ONNX Runtime for optimized inference

import onnxruntime as ort

# Convert model to ONNX for faster inference
session = ort.InferenceSession(
    "ranking_model.onnx",
    providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)

# Inference
outputs = session.run(
    ['watch_probability'],
    {
        'user_id': user_ids,
        'item_id': item_ids,
        'context_features': context_features
    }
)

5.3 A/B Testing Framework

class ABTestFramework:
    def __init__(self, experiments_config):
        self.experiments = experiments_config
        
    def assign_variant(self, user_id, experiment_id):
        """Deterministic user-to-variant assignment"""
        hash_value = hash(f"{user_id}:{experiment_id}")
        variant_index = hash_value % 100
        
        # Get variant boundaries from experiment config
        variants = self.experiments[experiment_id]['variants']
        cumulative = 0
        for variant in variants:
            cumulative += variant['traffic_percentage']
            if variant_index < cumulative:
                return variant['name']
        return variants[-1]['name']
    
    def log_impression(self, user_id, experiment_id, variant, 
                       item_id, metrics):
        """Log experiment impression for analysis"""
        event = {
            'user_id': user_id,
            'experiment_id': experiment_id,
            'variant': variant,
            'item_id': item_id,
            'metrics': metrics,
            'timestamp': datetime.utcnow().isoformat()
        }
        # Send to Kafka for offline analysis
        kafka_producer.send('experiment-events', event)

6. Monitoring and Observability

6.1 Key Metrics to Monitor

class RecommendationMetrics:
    """Comprehensive metrics for recommendation system monitoring"""
    
    # Online serving metrics
    ONLINE_METRICS = [
        'p50_latency_ms',
        'p95_latency_ms', 
        'p99_latency_ms',
        'qps',
        'error_rate',
        'cache_hit_rate',
        'model_inference_time_ms'
    ]
    
    # Recommendation quality metrics
    QUALITY_METRICS = [
        'click_through_rate',
        'watch_through_rate',
        'avg_watch_time',
        'user_satisfaction_score',
        'catalog_coverage',
        'diversity_score',
        'novelty_score',
        'serendipity_score'
    ]
    
    # Business metrics
    BUSINESS_METRICS = [
        'content_hours_streamed',
        'subscriber_retention_rate',
        'churn_rate',
        'revenue_per_user'
    ]

6.2 Monitoring Dashboard

# Grafana dashboard configuration (as code)
dashboard_config = {
    "panels": [
        {
            "title": "Recommendation Latency",
            "targets": [{
                "expr": "histogram_quantile(0.99, rec_latency_seconds_bucket)",
                "legendFormat": "p99 latency"
            }]
        },
        {
            "title": "CTR by Recommendation Surface",
            "targets": [{
                "expr": "sum(rate(rec_clicks_total[5m])) by (surface) / sum(rate(rec_impressions_total[5m])) by (surface)",
                "legendFormat": "{{surface}}"
            }]
        },
        {
            "title": "Model Prediction Distribution",
            "targets": [{
                "expr": "histogram_quantile(0.5, rec_prediction_score_bucket)",
                "legendFormat": "Median prediction score"
            }]
        }
    ]
}

6.3 Data Drift Detection

from scipy import stats
import numpy as np

class DataDriftDetector:
    def __init__(self, reference_data, window_size=10000):
        self.reference_data = reference_data
        self.window_size = window_size
        
    def detect_drift(self, new_data, feature_name, threshold=0.05):
        """
        Detect data drift using Kolmogorov-Smirnov test
        """
        ref_values = self.reference_data[feature_name]
        new_values = new_data[feature_name][-self.window_size:]
        
        # KS test
        ks_statistic, p_value = stats.ks_2samp(ref_values, new_values)
        
        # PSI (Population Stability Index)
        psi = self._calculate_psi(ref_values, new_values)
        
        return {
            'ks_statistic': ks_statistic,
            'p_value': p_value,
            'psi': psi,
            'drift_detected': p_value < threshold or psi > 0.2
        }
    
    def _calculate_psi(self, reference, current, n_bins=10):
        """Calculate Population Stability Index"""
        # Create bins from reference data
        bins = np.percentile(reference, np.linspace(0, 100, n_bins + 1))
        
        # Calculate distributions
        ref_dist = np.histogram(reference, bins=bins)[0] / len(reference)
        cur_dist = np.histogram(current, bins=bins)[0] / len(current)
        
        # Avoid division by zero
        ref_dist = np.clip(ref_dist, 0.001, None)
        cur_dist = np.clip(cur_dist, 0.001, None)
        
        # PSI formula
        psi = np.sum((cur_dist - ref_dist) * np.log(cur_dist / ref_dist))
        return psi

7. Scale Considerations and Trade-offs

7.1 Horizontal Scaling Strategy

┌─────────────────────────────────────────────────────────────────┐
│                    SCALING DIMENSIONS                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Users (260M+)                                                  │
│  └── Shard by user_id hash across recommendation services      │
│                                                                 │
│  Content (100K+ titles)                                         │
│  └── Shard item embeddings across FAISS indices                │
│                                                                 │
│  Features (1000+ dimensions)                                    │
│  └── Partition feature store by feature groups                 │
│                                                                 │
│  Model Complexity                                               │
│  └── Model parallelism for large embedding tables              │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

7.2 Cost vs Performance Trade-offs

7.3 Handling Cold Start

New User Cold Start:

class ColdStartHandler:
    def get_recommendations(self, new_user):
        # Strategy 1: Popularity-based
        popular_items = self.get_global_popularity()
        
        # Strategy 2: Demographic-based
        demographic_recs = self.get_by_demographics(
            new_user.age, new_user.location
        )
        
        # Strategy 3: Onboarding preferences
        if new_user.genre_preferences:
            genre_recs = self.get_by_genres(new_user.genre_preferences)
        
        # Strategy 4: Contextual bandits for exploration
        exploration_items = self.contextual_bandit.suggest(
            context=new_user.context
        )
        
        return self.merge_strategies(
            popular_items, demographic_recs, 
            genre_recs, exploration_items
        )

New Content Cold Start:

class NewContentHandler:
    def __init__(self):
        self.content_encoder = ContentEncoder()
        
    def get_initial_embeddings(self, new_content):
        # Use content metadata for initial embedding
        text_embedding = self.content_encoder.encode_text(
            new_content.title, new_content.description
        )
        visual_embedding = self.content_encoder.encode_thumbnail(
            new_content.thumbnail_url
        )
        
        # Find similar existing content
        similar_items = self.ann_index.search(
            np.concatenate([text_embedding, visual_embedding]),
            k=100
        )
        
        # Bootstrap engagement predictions from similar items
        initial_ctr = self.predict_from_similar(similar_items)
        
        return {
            'embedding': np.concatenate([text_embedding, visual_embedding]),
            'initial_ctr': initial_ctr,
            'exploration_weight': 0.3  # High exploration for new content
        }

7.4 Real-time vs Batch Trade-offs

Real-time Features:
+ Immediate adaptation to user behavior
+ Better relevance for time-sensitive content
- Higher infrastructure cost
- More complex implementation
- Potential latency issues

Batch Features:
+ Lower infrastructure cost
+ Simpler implementation and debugging
+ More stable predictions
- Slower adaptation to changing preferences
- Can't capture session-level context
- Stale recommendations during peak hours

Hybrid Approach (Recommended):
- Batch: User profiles, item embeddings, historical aggregates
- Near-real-time: Session features, recent interactions (1-5 min delay)
- Real-time: Current context, device, time-of-day features

8. Advanced Topics

8.1 Multi-objective Optimization

In production, you're optimizing for multiple competing objectives:

class ParetoOptimalRanker:
    """Rank items considering multiple objectives"""
    
    def rank(self, candidates, user_context):
        scores = {}
        for item in candidates:
            scores[item.id] = {
                'relevance': self.predict_relevance(item, user_context),
                'diversity': self.compute_diversity(item, user_context.seen_items),
                'novelty': self.compute_novelty(item, user_context),
                'freshness': self.compute_freshness(item),
                'business_value': self.compute_business_value(item)
            }
        
        # Pareto-optimal selection
        pareto_front = self.find_pareto_optimal(scores)
        
        # Re-rank using scalarization
        final_ranking = self.scalarize_and_sort(
            pareto_front,
            weights={'relevance': 0.6, 'diversity': 0.15, 
                    'novelty': 0.1, 'freshness': 0.1, 
                    'business_value': 0.05}
        )
        
        return final_ranking

8.2 Exploration vs Exploitation

class ThompsonSamplingExplorer:
    """Thompson Sampling for exploration-exploitation"""
    
    def __init__(self, n_items):
        self.alpha = np.ones(n_items)  # Success counts
        self.beta = np.ones(n_items)   # Failure counts
        
    def select_item(self, user_context, n_recommendations=10):
        # Sample from Beta distributions
        samples = np.random.beta(self.alpha, self.beta)
        
        # Select top-k sampled items
        selected = np.argsort(samples)[-n_recommendations:]
        
        return selected
    
    def update(self, item_id, was_watched):
        """Update beliefs based on user interaction"""
        if was_watched:
            self.alpha[item_id] += 1
        else:
            self.beta[item_id] += 1

8.3 Position Bias in Training

class PositionBiasCorrector:
    """Correct for position bias in logged data"""
    
    def __init__(self, position_propensity):
        self.propensity = position_propensity
        
    def compute_importance_weights(self, positions):
        """
        Inverse propensity scoring for position bias correction
        P(click|position) = P(click|relevant) * P(relevant) / P(position)
        """
        return 1.0 / self.propensity[positions]
    
    def weighted_loss(self, predictions, labels, positions):
        weights = self.compute_importance_weights(positions)
        return tf.reduce_mean(
            weights * tf.keras.losses.binary_crossentropy(labels, predictions)
        )

9. Implementation Roadmap

Phase 1: MVP (Weeks 1-4)

• Basic collaborative filtering for candidate generation
• Simple popularity-based fallback
• Batch feature computation
• Basic A/B testing framework

Phase 2: Foundation (Weeks 5-8)

• Two-tower model for candidate generation
• Feature store implementation (Redis + S3)
• Real-time feature pipeline (Kafka + Flink)
• Monitoring dashboard

Phase 3: Advanced (Weeks 9-12)

• Deep ranking model with cross-features
• Multi-task learning
• Advanced A/B testing with interleaving
• Position bias correction

Phase 4: Optimization (Weeks 13-16)

• Exploration strategies (Thompson Sampling)
• Multi-objective optimization
• Advanced monitoring (drift detection, feedback loops)
• Cost optimization

10. Summary and Key Takeaways

Architecture Recap

1. Three-stage funnel: Candidate generation → Ranking → Post-processing
2. Hybrid data pipeline: Batch + streaming + real-time
3. Multi-model approach: Different models for different stages
4. Feature store: Centralized feature management

Key Metrics

• Online: Latency (p50, p95, p99), QPS, error rate
• Quality: CTR, watch time, user satisfaction
• Business: Content hours, retention, revenue

Common Interview Mistakes

1. Jumping to deep learning without discussing simpler baselines
2. Ignoring cold start problem
3. Not discussing monitoring and observability
4. Forgetting about A/B testing framework
5. Not considering cost vs performance trade-offs

Further Reading

• "Deep Neural Networks for YouTube Recommendations" (Covington et al., 2016)
• "Wide & Deep Learning for Recommender Systems" (Cheng et al., 2016)
• "The Netflix Recommender System: Algorithms, Business Value, and Innovation"
• "Artificial Intelligence at Scale" (Netflix Tech Blog)
• "Recommendations: Beyond Click-Through Rate" (Google Research)