Project 2: End-to-End ML Pipeline
Build a production-grade ML pipeline that ingests raw data, engineers features, trains models, evaluates performance, and outputs deployment-ready artifacts.
Pipeline Architecture
1. Data Ingestion and Validation
Schema Enforcement
Data Drift Detection
2. Feature Engineering Pipeline
Sklearn Pipeline Implementation
3. Model Training and Selection
Hyperparameter Search Space
4. Evaluation Framework
5. Experiment Tracking
6. Model Serialization
Project Structure
Key Takeaways
- Modularity: Each stage is independently testable and replaceable
- Reproducibility: Pipeline configs + versioned data = reproducible experiments
- Automation: Orchestrate with Airflow, Prefect, or Kubeflow Pipelines
- Monitoring: Track data drift, model performance, and system health post-deployment