ML Foundations
The Mathematical Backbone of Every ML Algorithm
Linear algebra, calculus, and probability form the foundation of all machine learning. Master these concepts to truly understand how algorithms work.
- Linear Algebra — Vectors, matrices, and the language of data
- Calculus — Derivatives and gradient descent for optimization
- Probability and Statistics — Bayes' theorem, distributions, and inference
"Mathematics is the language in which God has written the universe."
Prerequisites
Before diving in, make sure you're comfortable with:
- Basic Algebra — Solving equations, exponents, logarithms
- Functions — What they are, how to evaluate them
- High School Geometry — Lines, slopes, coordinates
- Python Basics — NumPy arrays, basic operations
Learning Objectives
After completing this tutorial, you will be able to:
- Perform vector and matrix operations (addition, dot product, multiplication)
- Understand and compute derivatives and gradients
- Explain gradient descent and how it optimizes model parameters
- Apply Bayes' theorem to real-world classification problems
- Identify key probability distributions used in ML
- Connect each mathematical concept to its ML application
Math Foundations for Machine Learning
Math is the language of machine learning. This tutorial covers the essential math you need — with clear explanations, visual intuitions, and Python code.
Linear Algebra
Vectors and Vector Operations
Matrices
Matrix Operations in ML
Calculus
Derivatives and Gradients
Gradient Descent
Partial Derivatives and the Gradient
Chain Rule
Probability
Probability Axioms
Conditional Probability and Bayes' Theorem
Distributions
Expectation and Variance
Common Mistakes & How to Avoid Them
Mistake 1: Confusing dot product with matrix multiplication
- Problem: Dot product gives a scalar (), matrix multiplication gives a matrix ()
- Solution: Remember dimensions — dot product is (1×n)(n×1) → scalar; matmul is (m×k)(k×n) → (m×n)
Mistake 2: Forgetting to transpose in matrix operations
- Problem: requires to be (N×d) and to be (d×1). Mismatched dimensions cause errors.
- Solution: Always check: (N×d)(d×1) = (N×1) ✓
Mistake 3: Applying gradient descent without feature scaling
- Problem: If features have different scales (e.g., age 0-100 vs income 0-1,000,000), gradient descent oscillates and converges slowly
- Solution: Standardize features: before applying gradient descent
Mistake 4: Ignoring numerical precision in log/probability computations
- Problem: Computing or multiplying many small probabilities causes underflow
- Solution: Use log-space: , add small ε to avoid log(0)
Mistake 5: Confusing correlation with causation
- Problem:
- Solution: Correlation measures linear association only. Causal inference requires additional assumptions and experiments.
Interview Questions
Q1: Why is the dot product important in ML? A: The dot product computes weighted sums — the core operation in neural networks (), measures similarity between vectors (cosine similarity), and enables kernel methods in SVMs.
Q2: What is the gradient and why does it matter for optimization? A: The gradient is the vector of partial derivatives pointing in the direction of steepest ascent. For ML optimization, gives the direction of steepest decrease in loss, enabling us to iteratively improve model parameters.
Q3: Explain Bayes' theorem with a real example. A: . If 30% of emails are spam, and "free" appears in 80% of spam but 5% of ham, then . Bayes lets us update our belief given new evidence.
Q4: Why do we use the normal distribution so much in ML? A: The Central Limit Theorem states that the sum of many independent random variables tends toward a normal distribution, regardless of the original distribution. This makes it a natural choice for modeling errors, noise, and feature distributions.
Q5: What is the curse of dimensionality and how does it relate to linear algebra? A: As dimension increases, the volume of space grows exponentially (). In high dimensions, all points become approximately equidistant (distance concentration), making distance-based methods like KNN unreliable. PCA and feature selection help mitigate this.
Q6: Why is the chain rule essential for training neural networks? A: Neural networks are compositions of functions (layers). The chain rule enables backpropagation — efficiently computing for every weight by propagating gradients backward through the network in O(N) time, where N is the number of connections.
Q7: What is the difference between L1 and L2 norms? A: L1 norm promotes sparsity (used in Lasso regularization). L2 norm promotes small, distributed weights (used in Ridge regularization). L1 is also used in Manhattan distance; L2 in Euclidean distance.
Practice Exercise
Challenge: Implement Gradient Descent from Scratch
Implement gradient descent to find the minimum of :
import numpy as np
# Define the function and its gradient
def f(x, y):
return (x - 3)**2 + (y + 2)**2
def gradient(x, y):
df_dx = 2 * (x - 3) # partial derivative w.r.t. x
df_dy = 2 * (y + 2) # partial derivative w.r.t. y
return np.array([df_dx, df_dy])
# Gradient descent
lr = 0.1 # learning rate
params = np.array([0.0, 0.0]) # start at origin
history = [params.copy()]
for i in range(20):
grad = gradient(params[0], params[1])
params = params - lr * grad
history.append(params.copy())
loss = f(params[0], params[1])
if i % 5 == 0:
print(f"Step {i:2d}: x={params[0]:.4f}, y={params[1]:.4f}, loss={loss:.6f}")
print(f"\nConverged to: ({params[0]:.4f}, {params[1]:.4f})")
print(f"True minimum: (3, -2)")
Bonus challenges:
- Experiment with different learning rates (0.01, 0.1, 0.5, 1.1). What happens?
- Implement momentum-based gradient descent and compare convergence speed
- Plot the contour of the loss function with the optimization trajectory
Comparison Table
Mathematical Concepts Comparison
| Concept | Definition | ML Application | Key Formula |
|---|---|---|---|
| Dot Product | Sum of element-wise products | Neural network forward pass | |
| Derivative | Rate of change at a point | Optimization direction | |
| Gradient | Vector of partial derivatives | Steepest descent direction | |
| Bayes' Theorem | Posterior from prior + evidence | Probabilistic classification | |
| Variance | Spread of distribution | Model stability / overfitting |
Key Formulas Reference
| Formula | Expression | Context |
|---|---|---|
| Vector Addition | Element-wise sum | |
| Dot Product | Similarity, neural nets | |
| Matrix Multiply | Linear transformations | |
| Power Rule | Derivative computation | |
| Gradient Descent | Model optimization | |
| Chain Rule | Backpropagation | |
| Bayes' Theorem | Probabilistic inference | |
| Normal PDF | Error modeling |
Key Takeaways
What to Learn Next
-> What is Machine Learning? The complete introduction to ML — concepts, types, and workflow.
-> Linear Regression From scatter plots to predictions — the simplest ML algorithm.
-> Logistic Regression Classification with probability — from linear to sigmoid.
-> Dimensionality Reduction Reduce features while preserving information with PCA and t-SNE.
-> Regularization Prevent overfitting with Ridge, Lasso, and Elastic Net.
-> KNN Instance-based learning where your neighbors tell the story.