Math Foundations for Deep Learning

Deep learning is built on linear algebra, calculus, and probability. This tutorial covers the essential math you need to understand how neural networks learn.

See our Math Linear Algebra tutorial and Math Calculus tutorial for broader mathematical foundations.

Linear Algebra

Vectors and Matrices

DfVector and Matrix Operations

Dot Product: $\mathbf{u} \cdot \mathbf{v} = \sum_i u_i v_i = \|\mathbf{u}\| \|\mathbf{v}\| \cos\theta$
Matrix Multiplication: $\mathbf{C} = \mathbf{A}\mathbf{B}$ where $C_{ij} = \sum_k A_{ik} B_{kj}$
Norm: $\|\mathbf{x}\|_p = \left(\sum_i |x_i|^p\right)^{1/p}$
Transpose: $(\mathbf{A}^T)_{ij} = A_{ji}$

Matrix Multiplication

C_{ij} = \sum_{k=1}^{n} A_{ik} \cdot B_{kj}

Here,

$A$ =Input matrix of shape (m x n)
$B$ =Input matrix of shape (n x p)
$C$ =Output matrix of shape (m x p)
$n$ =Inner dimension (must match)

Eigenvalues and Eigenvectors

DfEigenvalue Decomposition

For a square matrix $\mathbf{A}$ , an eigenvector $\mathbf{v}$ and eigenvalue $\lambda$ satisfy:

\mathbf{A}\mathbf{v} = \lambda \mathbf{v}

The eigendecomposition $\mathbf{A} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^T$ reveals the matrix's action: stretching along eigenvector directions by eigenvalue factors. This is critical for understanding:

PCA: Principal components are eigenvectors of the covariance matrix
Hessian analysis: Eigenvalues indicate curvature directions
Spectral initialization: Eigenvectors of weight matrices

\mathbf{A} = \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^T \quad \text{where} \quad \mathbf{\Lambda} = \text{diag}(\lambda_1, \ldots, \lambda_n)

Singular Value Decomposition (SVD)

DfSVD

Every matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$ can be decomposed as:

\mathbf{A} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^T

where $\mathbf{U} \in \mathbb{R}^{m \times m}$ and $\mathbf{V} \in \mathbb{R}^{n \times n}$ are orthogonal, and $\mathbf{\Sigma}$ is diagonal with singular values. SVD is used in weight pruning, low-rank approximation, and understanding network expressivity.

Calculus for Deep Learning

Gradients

DfGradient

The gradient of a scalar function $f: \mathbb{R}^n \to \mathbb{R}$ is the vector of partial derivatives:

\nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \\ \vdots \\ \frac{\partial f}{\partial x_n} \end{bmatrix}

The gradient points in the direction of steepest ascent. Gradient descent minimizes $f$ by moving in the direction $-\nabla f$ .

Gradient of a Function

\nabla f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{partial x_2} \\ \vdots \\ \frac{\partial f}{\partial x_n} \end{bmatrix}

Here,

$f$ =Scalar-valued function
$x$ =Input vector
$\nabla f$ =Gradient vector (same shape as x)

The Chain Rule

DfChain Rule

For composite functions, the chain rule gives:

\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x)

For multivariate functions:

\frac{\partial z}{\partial x_i} = \sum_j \frac{\partial z}{\partial y_j} \cdot \frac{\partial y_j}{\partial x_i}

This is the foundation of backpropagation — the algorithm that trains all neural networks.

Chain Rule for Multivariate Functions

\frac{\partial z}{\partial x_i} = \sum_{j} \frac{\partial z}{\partial y_j} \cdot \frac{\partial y_j}{\partial x_i}

Here,

$z$ =Output variable
$y_j$ =Intermediate variables
$x_i$ =Input variables

ThChain Rule for Deep Networks (Composition)

For a deep network $f_L \circ f_{L-1} \circ \cdots \circ f_1$ , the gradient of the loss $\mathcal{L}$ with respect to parameters $\theta^{(l)}$ in layer $l$ is:

\frac{\partial \mathcal{L}}{\partial \theta^{(l)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{h}^{(L)}} \cdot \prod_{k=l+1}^{L} \frac{\partial \mathbf{h}^{(k)}}{\partial \mathbf{h}^{(k-1)}} \cdot \frac{\partial \mathbf{h}^{(l)}}{\partial \theta^{(l)}}

Each factor $\frac{\partial \mathbf{h}^{(k)}}{\partial \mathbf{h}^{(k-1)}}$ is the Jacobian of layer $k$ . The product of these Jacobians causes vanishing or exploding gradients when network depth $L$ is large.

The Hessian

DfHessian Matrix

The Hessian is the matrix of second-order partial derivatives:

\mathbf{H}_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}

It captures the curvature of $f$ . Positive definite Hessians indicate local minima; indefinite Hessians indicate saddle points.

Hessian Matrix

\mathbf{H} = \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \cdots \\ \frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{partial x_2^2} & \cdots \\ \vdots & \vdots & \ddots \end{bmatrix}

Here,

$\mathbf{H}$ =Hessian matrix (n x n)
$f$ =Scalar function to differentiate
$x_i$ =Input variable i

ℹ️ Hessian and Optimization

Positive definite Hessian ( $\mathbf{H} \succ 0$ ): Local minimum
Negative definite Hessian ( $\mathbf{H} \prec 0$ ): Local maximum
Indefinite Hessian: Saddle point (common in high-dimensional spaces)
Largest eigenvalue of Hessian = maximum curvature = affects step size
Second-order optimizers (L-BFGS) use Hessian info but are expensive for large models

Probability Theory

Key Distributions

DfGaussian Distribution

The Gaussian (normal) distribution is fundamental to deep learning:

p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)

Weights are often initialized from Gaussians
Batch normalization assumes Gaussian activations
VAEs model the latent space as Gaussian

DfBernoulli Distribution

The Bernoulli distribution models binary outcomes:

p(x) = \begin{cases} p & \text{if } x = 1 \\ 1-p & \text{if } x = 0 \end{cases}

Used for binary classification, dropout masks, and binary cross-entropy loss.

Information Theory

DfCross-Entropy

Cross-entropy measures the difference between two probability distributions:

H(p, q) = -\sum_x p(x) \log q(x)

In classification, $p$ is the true label distribution and $q$ is the predicted distribution. Minimizing cross-entropy is equivalent to maximizing the likelihood of the correct class.

\mathcal{L}_{\text{CE}} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^{C} y_{ic} \log(\hat{y}_{ic})

DfKL Divergence

KL divergence measures how one distribution diverges from another:

D_{\text{KL}}(p \| q) = \sum_x p(x) \log \frac{p(x)}{q(x)}

It is always non-negative and equals zero only when $p = q$ . Used in VAEs, knowledge distillation, and distribution matching.

Practical Example: Gradient Computation

📝Example: Manual Gradient Computation

import torch

# Define a simple computation: f(x, y) = (x^2 + y^2) * exp(-(x^2 + y^2))
# This is a "Mexican hat" function

x = torch.tensor(1.0, requires_grad=True)
y = torch.tensor(2.0, requires_grad=True)

# Forward pass
r_sq = x**2 + y**2
z = r_sq * torch.exp(-r_sq)

# Backward pass (computes gradients via chain rule)
z.backward()

# Analytical gradients:
# df/dx = (2x - 2x(x^2 + y^2)) * exp(-(x^2 + y^2))
# df/dy = (2y - 2y(x^2 + y^2)) * exp(-(x^2 + y^2))

print(f"f(1, 2) = {z.item():.6f}")
print(f"df/dx = {x.grad.item():.6f}")
print(f"df/dy = {y.grad.item():.6f}")

# Verify with torch.autograd.grad
grad_x, grad_y = torch.autograd.grad(
    z, [x, y], create_graph=True
)
print(f"\nAutograd df/dx: {grad_x.item():.6f}")
print(f"Autograd df/dy: {grad_y.item():.6f}")

📝Example: Hessian-Vector Product

import torch

x = torch.tensor([2.0, 3.0], requires_grad=True)

# Quadratic function: f(x) = 0.5 * x^T H x
H = torch.tensor([[4.0, 1.0], [1.0, 3.0]])
f = 0.5 * x @ H @ x

# First-order gradient (Jacobian-vector product)
grad = torch.autograd.grad(f, x, create_graph=True)[0]
print(f"Gradient: {grad}")

# Hessian-vector product: H @ v
v = torch.tensor([1.0, 1.0])
hvp = torch.autograd.grad(grad, x, grad_outputs=v, retain_graph=True)[0]
print(f"Hessian-vector product: {hvp}")
print(f"Direct H @ v: {H @ v}")

Summary

📋Summary: Math Foundations for Deep Learning

Linear algebra: Vectors, matrices, eigendecomposition, SVD — underpin all neural network operations
Chain rule: The backbone of backpropagation — enables efficient gradient computation
Hessian: Captures curvature — positive definite for minima, indefinite for saddle points
Gaussian distribution: Weight initialization, batch normalization, VAEs
Cross-entropy: The standard loss for classification — equivalent to maximum likelihood
KL divergence: Measures distribution differences — used in VAEs, distillation
Matrix calculus: Extends gradient computation to matrix-valued functions (weight matrices)

Practice Exercises

Linear Algebra: Given $\mathbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 3 \end{bmatrix}$ , compute its eigenvalues and eigenvectors. Verify that $\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$ .
Chain Rule: Let $f(x) = \sin(x^2)$ and $g(x) = e^{-x^2}$ . Compute $\frac{d}{dx} f(g(x))$ using the chain rule. Verify with PyTorch autograd.
Hessian: Compute the Hessian of $f(x, y) = x^4 + y^4 - 4xy + 1$ . Find all critical points and classify them (min/max/saddle).
Probability: Derive the gradient of the negative log-likelihood for a Gaussian distribution with respect to $\mu$ and $\sigma$ . Implement it in PyTorch.
Coding: Implement a function that computes the full Hessian matrix of a scalar function using PyTorch's autograd. Test it on a 2D quadratic function.

Math Foundations for Deep Learning — Linear Algebra, Calculus & Probability

Math Foundations for Deep Learning

Linear Algebra

Vectors and Matrices

DfVector and Matrix Operations

Matrix Multiplication

Eigenvalues and Eigenvectors

DfEigenvalue Decomposition

Singular Value Decomposition (SVD)

DfSVD

Calculus for Deep Learning

Gradients

DfGradient

Gradient of a Function

The Chain Rule

DfChain Rule

Chain Rule for Multivariate Functions

ThChain Rule for Deep Networks (Composition)

The Hessian

DfHessian Matrix

Hessian Matrix

Probability Theory

Key Distributions

DfGaussian Distribution

DfBernoulli Distribution

Information Theory

DfCross-Entropy

DfKL Divergence

Practical Example: Gradient Computation

📝Example: Manual Gradient Computation

📝Example: Hessian-Vector Product

Summary

📋Summary: Math Foundations for Deep Learning

Practice Exercises

Need Expert Deep Learning Help?