Why This Matters

ML pipelines are the backbone of production machine learning systems. Without robust pipelines:

• Manual processes lead to errors and inconsistency
• Reproducibility becomes impossible
• Scaling to multiple models is unmanageable
• Collaboration between teams breaks down

Pipeline Components Overview

ML Pipeline Stages Architecture

Pipeline Implementation

Orchestration with Apache Airflow

ML pipelines are typically orchestrated using workflow management tools like Apache Airflow or Kubeflow Pipelines. An Airflow DAG defines the sequence of tasks for data extraction, feature engineering, model training, evaluation, and deployment. Tasks are connected via dependencies (e.g., extract_task >> feature_task >> train_task >> deploy_task), ensuring each stage completes before the next begins. Airflow handles scheduling, retries, alerting, and monitoring.

Orchestration with Kubeflow Pipelines

Kubeflow Pipelines provide a Kubernetes-native approach to ML pipeline orchestration. Components are defined as reusable YAML specs, and pipelines are constructed using the KFP SDK. Conditional logic (e.g., only deploy if accuracy > 0.85) enables quality gates. Pipelines are compiled and submitted to a Kubeflow server for execution.

Pipeline Patterns

Pattern 1: Linear Pipeline

A simple linear pipeline executes steps sequentially: extract -> transform -> train -> evaluate -> deploy. Each step takes the output of the previous step as input.

Pattern 2: DAG Pipeline

Complex pipelines use Directed Acyclic Graphs to model dependencies. Independent tasks (e.g., extracting users and items) run in parallel, while dependent tasks wait for their prerequisites to complete.

Pattern 3: Parallel Pipeline

Parallel pipelines use concurrent execution to speed up independent tasks. Multiple tasks run simultaneously using thread or process pools, and results are collected as they complete.

Data Validation with Great Expectations

Data validation ensures input data meets expected quality standards before entering the pipeline. Expectation suites define rules such as: column values must not be null, values must be unique, and numeric values must fall within expected ranges. Checkpoints run these validations and report success or failure.

Monitoring Pipeline Health

Pipeline monitoring tracks key operational metrics including total pipeline runs, failure counts, execution duration, and data quality scores. A monitoring context manager wraps pipeline execution to automatically capture metrics, log failures, and track data quality throughout the pipeline lifecycle.

Sizing the Pipeline Window

A pipeline that must publish recommendations by 06:00 owns a wall-clock budget, and every stage spends from it:

With measured stage times of 40 minutes ingestion, 70 minutes feature computation, 90 minutes training, 30 minutes evaluation, and 20 minutes deployment, the run needs 250 minutes. Starting at 01:50 lands at 06:00 exactly, which is too tight: one retry of the training stage alone adds 90 minutes and misses the deadline. Teams that plan for retries reserve a window of about 360 minutes for the same 250 minutes of work, keeping utilization near 70 percent so a re-run still lands on time.

Throughput is the second constraint. Scoring 250 million rows inside a 4 hour window on 40 workers sets a per-worker rate:

So each worker must sustain about 434 rows per second for four hours straight. If the measured rate is 300, you either add 45 percent more workers, shrink the feature set, or move to incremental processing that only scores the partitions that changed.

Retries do not make a pipeline reliable on their own. With tasks, a attempt budget, and a chance that any single attempt fails transiently, the probability that a whole run still fails is:

That is 2.5 percent of runs, or roughly seven failed nights per month at 30 runs. The fix is not more retries; it is checkpoints on the expensive stages plus alerting on the deadline itself, so the remaining window is what pages you.

Freshness has its own arithmetic. A consumer reading hourly batches waits half an interval on average, then the pipeline time, then the deployment:

Features are therefore about 75 minutes stale at the moment a model scores with them. If the product requirement is 15 minutes, no amount of DAG tuning helps: the schedule interval has to shrink, which is the decision that pushes you toward a streaming feature pipeline.

Interview Q&A

Answer:

1. Schema Registry: Use a schema registry (e.g., AWS Glue, Confluent Schema Registry) to track schema versions
2. Backward Compatibility: Always design schemas to be backward compatible
3. Feature Versioning: Version features alongside data
4. Graceful Degradation: Handle missing/changed features gracefully

A schema manager compares the current schema against the target schema. If they are incompatible, the manager adapts the data to the new schema format, ensuring downstream components receive properly structured data.

Answer:

1. Version Control: Track code, data, and configurations
2. Pinned Dependencies: Use exact version numbers
3. Random Seeds: Set all random seeds
4. Environment Isolation: Use containers
5. Experiment Tracking: Log all experiments

Reproducibility requires setting random seeds across all frameworks (Python, NumPy, PyTorch) and ensuring deterministic behavior in CUDA operations.

Answer:

Frequently Asked Questions

Why not put the whole workflow in one big script?

A single script hides its dependencies, cannot restart halfway, and gives the scheduler nothing to retry or parallelize. Split into tasks with declared dependencies and independent branches run in parallel, a failure retries only the affected task, and every run records which inputs produced which outputs. The DAG is also where you attach data quality gates between stages.

How much margin should a nightly job carry?

Plan to fill roughly 70 percent of the window. A run that exactly fills 250 minutes has zero retry capacity, so the first transient failure blows the deadline. Reserving about 360 minutes for 250 minutes of work absorbs one complete re-run of the slowest stage, which is almost always training.

When is a DAG the wrong pattern?

When the data never stops arriving and freshness matters more than completeness. A micro-batch stream computes features every minute or five, keeps the freshness lag near the interval length instead of 75 minutes, and leaves the heavy nightly DAG for training and backfills. Use both: streaming for serving features, scheduled DAGs for model training.

Quiz

Next Steps

• Module 3: Feature Store Design - Deep dive into feature management
• Module 4: Model Serving Architecture - Learn to serve models at scale
• Practice: Design a complete ML pipeline for a recommendation system

Continue to Feature Store Design ->