ML Foundations
The Science of Getting Computers to Learn from Data
Machine learning is transforming every industry — from healthcare to finance to autonomous vehicles. Understanding the fundamentals is the first step to building intelligent systems.
- Supervised Learning — Learn from labeled data to make predictions
- Unsupervised Learning — Discover hidden patterns in unlabeled data
- The ML Workflow — A systematic approach from problem definition to deployment
"Machine learning is the last invention that humanity will ever need to make."
Prerequisites
Before diving in, make sure you're comfortable with:
- Basic Python — Variables, loops, functions, and libraries (NumPy, Pandas)
- High School Math — Algebra, basic statistics (mean, median, standard deviation)
- Data Literacy — What CSV files are, what rows/columns represent
- Terminal Basics — Running Python scripts and installing packages
Learning Objectives
After completing this tutorial, you will be able to:
- Define machine learning and distinguish it from traditional programming
- Identify the three main types of ML: supervised, unsupervised, and reinforcement learning
- Describe the complete ML workflow from problem definition to deployment
- Explain bias-variance tradeoff and why it matters for model selection
- Recognize common ML algorithms and their appropriate use cases
- Understand overfitting, underfitting, and strategies to combat them
- Appreciate the role of data quality in building effective ML systems
What is Machine Learning? — Complete Introduction
Machine Learning is the science of getting computers to learn from data without being explicitly programmed. This tutorial provides a comprehensive foundation for your entire ML journey.
What is Machine Learning?
Traditional Programming vs Machine Learning
How ML reverses traditional programming: The top half shows traditional programming: a human explicitly writes rules (if-else statements) that transform input data into outputs. For email spam filtering, you'd write rules like "if email contains 'free money', mark as spam." The bottom half shows the ML approach: instead of writing rules, you provide examples (labeled emails) and the algorithm automatically discovers the rules. The red "Learned Rules (Model)" box represents what the ML algorithm produces — a mathematical function that maps inputs to outputs. The text at the bottom summarizes the paradigm shift: traditional = Data + Rules → Output; ML = Data + Output → Rules. This is powerful because the learned rules can capture patterns too complex for humans to specify manually — like recognizing spam based on thousands of subtle features simultaneously.
Types of Machine Learning
Supervised Learning
Unsupervised Learning
Reinforcement Learning
ML Algorithm Taxonomy
Key Applications
Real-World Applications
Healthcare: Detecting Diabetic Retinopathy
Google's DeepMind trained a deep learning model to detect diabetic retinopathy from retinal fundus photographs. The model achieved performance comparable to ophthalmologists, with an AUC of 0.99. This enables early screening in areas with limited access to specialists.
Finance: JPMorgan's COIN System
JPMorgan's Contract Intelligence (COIN) platform uses ML to review commercial loan agreements. What previously took legal staff 360,000 hours annually now completes in seconds with higher accuracy, extracting 150 important data points from each document.
Retail: Amazon's Recommendation Engine
Amazon's ML-powered recommendation system drives 35% of total revenue. It analyzes purchase history, browsing behavior, and collaborative filtering to suggest products. The system processes hundreds of millions of customers in real-time.
Transportation: Waymo's Self-Driving Cars
Waymo's autonomous vehicles use a combination of computer vision, sensor fusion, and reinforcement learning. Their ML models process LiDAR, camera, and radar data simultaneously to make real-time driving decisions with superhuman safety records on highways.
NLP: GPT and Large Language Models
Large language models like GPT-4 use transformer architectures trained on trillions of tokens. These models demonstrate emergent capabilities including reasoning, code generation, and multi-language translation — all learned from statistical patterns in text data.
The ML Workflow
Key Concepts
Training, Validation, and Test Sets
Bias-Variance Decomposition
Overfitting vs Underfitting
Common ML Algorithms
Common Mistakes & How to Avoid Them
Mistake 1: Using test data for training
- Problem: Evaluating on test data during development leads to overfitting to the test set
- Solution: Always use a 3-way split (train/validation/test) or cross-validation
Mistake 2: Not handling missing data
- Problem: Dropping rows with missing values discards valuable information
- Solution: Use imputation (mean, median, KNN) or models that handle missing data natively
Mistake 3: Ignoring feature scaling
- Problem: Algorithms like KNN, SVM, and gradient descent are sensitive to feature magnitudes
- Solution: Standardize or normalize features before training distance-based models
Mistake 4: Not checking class balance
- Problem: 99% accuracy on 99/1 class split is meaningless
- Solution: Check class distribution, use F1/AUC-ROC instead of accuracy, apply resampling
Mistake 5: Overcomplicating from the start
- Problem: Jumping to deep learning before trying simple baselines
- Solution: Start with simple models (logistic regression, decision tree), then increase complexity
Mistake 6: Ignoring data leakage
- Problem: Using future information in training (e.g., including test-set features in training)
- Solution: Carefully separate train/test, use pipelines, validate temporal ordering
Mistake 7: Not performing EDA
- Problem: Building models without understanding the data leads to poor feature engineering
- Solution: Always explore distributions, correlations, and outliers before modeling
Interview Questions
Q1: What is the difference between supervised and unsupervised learning? A: Supervised learning uses labeled data to learn a mapping . Unsupervised learning finds hidden patterns in unlabeled data only. Examples: supervised = spam detection (with labels), unsupervised = customer segmentation (no labels).
Q2: What is overfitting and how do you prevent it? A: Overfitting is when a model memorizes training data including noise, performing well on training but poorly on test data. Prevention: regularization, cross-validation, more data, early stopping, simpler models, ensemble methods.
Q3: What is the bias-variance tradeoff? A: Error = Bias² + Variance + Noise. High bias = underfitting (model too simple). High variance = overfitting (model too complex). The goal is finding the sweet spot that minimizes total error by balancing both.
Q4: Why is accuracy a bad metric for imbalanced datasets? A: If 99% of data is class A, a model predicting "always A" gets 99% accuracy but is useless. Use precision, recall, F1-score, or AUC-ROC which account for class distribution.
Q5: What is cross-validation and why use it? A: Cross-validation splits data into K folds, trains on K-1, tests on 1, rotating K times. It provides more reliable performance estimates than a single train/test split and uses all data for both training and evaluation.
Q6: Explain the difference between generative and discriminative models. A: Discriminative models learn directly (e.g., logistic regression). Generative models learn and , then use Bayes' theorem (e.g., Naive Bayes). Discriminative models typically perform better for classification.
Q7: What steps would you take when building an ML model for the first time? A: (1) Define the problem and success metrics, (2) Explore and understand the data (EDA), (3) Clean and preprocess data, (4) Engineer relevant features, (5) Start with a simple baseline model, (6) Evaluate with appropriate metrics and cross-validation, (7) Iterate and improve, (8) Deploy and monitor.
Practice Exercise
Challenge: Your First ML Pipeline
Build a complete ML workflow using scikit-learn on the Iris dataset:
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report
import pandas as pd
import numpy as np
# Step 1: Load and explore the data
iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df['target'] = iris.target
print("Shape:", df.shape)
print("\nFirst 5 rows:")
print(df.head())
print("\nClass distribution:")
print(df['target'].value_counts())
print("\nBasic statistics:")
print(df.describe())
# Step 2: Split data
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
# Step 3: Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Step 4: Train multiple models
models = {
'Logistic Regression': LogisticRegression(max_iter=200),
'Decision Tree': DecisionTreeClassifier(max_depth=3)
}
for name, model in models.items():
# Cross-validation
cv_scores = cross_val_score(model, X_train_scaled, y_train, cv=5)
print(f"\n{name}:")
print(f" CV Accuracy: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")
# Train and evaluate on test set
model.fit(X_train_scaled, y_train)
test_score = model.score(X_test_scaled, y_test)
print(f" Test Accuracy: {test_score:.3f}")
# Detailed classification report
y_pred = model.predict(X_test_scaled)
print(f"\n Classification Report:")
print(classification_report(y_test, y_pred, target_names=iris.target_names))
# Step 5: Feature importance (Decision Tree)
dt_model = models['Decision Tree']
for name, imp in zip(iris.feature_names, dt_model.feature_importances_):
print(f" {name}: {imp:.3f}")
Bonus challenges:
- Try different values of
max_depthand plot accuracy vs depth - Add L1 and L2 regularization to Logistic Regression and compare
- Visualize the decision boundaries using
matplotlib
Comparison Table
ML Types Comparison
| Aspect | Supervised | Unsupervised | Reinforcement |
|---|---|---|---|
| Data | Labeled pairs | Unlabeled only | States, actions, rewards |
| Goal | Learn | Discover structure in | Maximize cumulative reward |
| Examples | Regression, Classification | Clustering, Dim. Reduction | Game AI, Robotics |
| Algorithms | Linear Reg, SVM, Trees, NN | K-Means, PCA, DBSCAN | Q-Learning, Policy Gradient |
| Feedback | Direct labels | None | Delayed rewards |
| Complexity | Moderate | Low-Moderate | High |
Key Formulas Reference
| Formula | Expression | Context |
|---|---|---|
| ML Definition | with | Mitchell, 1997 |
| Supervised Learning | Labeled data mapping | |
| Expected Loss | Model objective | |
| RL Discounted Return | Cumulative reward | |
| Bias-Variance | Error decomposition |
Key Takeaways
Further Reading
- "An Introduction to Statistical Learning" (ISLR) by James, Witten, Hastie, Tibshirani — Free, accessible introduction to ML
- "Pattern Recognition and Machine Learning" by Christopher Bishop — Comprehensive probabilistic perspective
- "The Elements of Statistical Learning" by Hastie, Tibshirani, Friedman — Advanced, mathematically rigorous (free PDF available)
- Andrew Ng's Machine Learning Specialization (Coursera) — Excellent video lectures and hands-on projects
- Google's Machine Learning Crash Course — Free, fast-paced introduction with TensorFlow exercises
- fast.ai Practical Deep Learning — Top-down approach, learn by building projects first
What to Learn Next
-> Math Foundations Master the essential math — vectors, matrices, derivatives, and probability.
-> Linear Regression The simplest and most fundamental ML algorithm for predicting continuous values.
-> Logistic Regression Classification with probability — from linear to sigmoid.
-> KNN Instance-based learning where your neighbors tell the story.
-> Decision Trees If-then rules that learn — the most interpretable algorithm.
-> Model Evaluation How to know if your model actually works — beyond accuracy.