DL Foundations
Backpropagation - How Neural Networks Actually Learn
Backpropagation is the algorithm that trains all neural networks. It efficiently computes the gradient of the loss with respect to every parameter using the chain rule, enabling gradient descent optimization.
- Forward Pass - Compute outputs layer by layer while storing intermediate values
- Backward Pass - Propagate error signals backward to compute parameter gradients
- Efficient Gradient Computation - Same complexity as forward pass, reusing intermediate computations
Backpropagation Algorithm - Forward Pass, Backward Pass and Computational Graphs
What You'll Learn
- 1Decompose a scalar expression into a computational graph and differentiate it node by node
- 2Write the forward and backward recurrences for an L layer network
- 3Explain why reverse mode costs about two forward passes for every gradient at once
- 4Diagnose vanishing and exploding gradients from the norm of the layer Jacobians
Backpropagation is the algorithm that enables neural networks to learn from data. It efficiently computes the gradient of the loss with respect to every parameter using the chain rule.
What Is Backpropagation?
Computational Graphs
Forward Pass
Backward Pass
Each recurrence costs one matrix multiply per layer plus an elementwise product, and every activation it touches was written to memory during the forward pass. That storage requirement, not the arithmetic, is the reason training a 7 billion parameter transformer needs tens of gigabytes of accelerator memory for activations alone.
The gradients then feed the update:
which is plain gradient descent. Momentum, Adam, and learning rate schedules only change how that one line is scaled.
Gradient Flow Through Layers
PyTorch Autograd
Practical Considerations
Gradient clipping in the table is the norm cap:
A typical is 1.0 for transformer training, and the cap costs one norm reduction per step. It stops a rare bad batch from writing a huge update over the careful progress of ten thousand good ones, which is the practical defense against exploding gradients even when initialization is correct.
Summary
- Backpropagation efficiently computes gradients using the chain rule
- The forward pass computes outputs and stores intermediates; the backward pass computes gradients
- Computational graphs represent the sequence of operations for automatic differentiation
- Vanishing/exploding gradients are fundamental challenges solved by architectural innovations
- PyTorch autograd implements reverse-mode automatic differentiation
Next: Activation Functions
Common Mistakes
Calling backward twice on the same graph
Fix: Accumulate gradients on purpose with retain_graph, or zero them with optimizer.zero_grad() between steps, otherwise you add the old gradient again.
Expecting gradients after an in-place operation on a leaf
Fix: In-place writes break the graph; use out-of-place ops or .detach() when you really mean to stop history.
Reading a vanishing gradient as a data problem first
Fix: Compute the norm of the gradient per layer; if it decays geometrically the cause is sigmoid or tanh plus deep chains, not your labels.
Best Practices
- Log the L2 norm of gradients per layer every hundred steps to catch decay or spikes early
- Verify backprop against finite differences on a two element toy model before trusting a new architecture
- Clip by global norm once per step rather than per parameter so direction is preserved
Key Takeaways
- Backprop is reverse-mode autodiff: one forward pass caches activations, one backward pass recovers every gradient
- The recurrences are delta(l) = W(l+1)^T delta(l+1) elementwise sigma prime and dL/dW(l) = delta(l) outer h(l-1)
- Reverse mode costs about 2x a forward pass for all n parameters, while forward mode would cost n passes
- Gradients scale as (max norm times max sigma prime) to the power L minus 1, which is the vanishing gradient problem
Frequently Asked Questions
Why is backpropagation not just the chain rule?
It is the chain rule, arranged so each intermediate derivative is computed once and stored. Naive differentiation of a deep network re-derives shared subexpressions repeatedly and blows up exponentially in depth; backprop visits each edge of the graph exactly once in reverse topological order, so the cost stays proportional to the forward pass.
What actually gets stored in memory during training?
Every activation that the backward pass will multiply: layer inputs for the weight gradients, and pre-activations for sigma prime. For a 12 layer, 4,096 wide transformer with batch 32 and sequence 2,048 those tensors run to multiple gigabytes, which is why gradient checkpointing recomputes segments of the forward pass instead of storing them.
How do you tell vanishing from exploding gradients in practice?
Plot the gradient norm per layer. Vanishing shows a monotone geometric decay toward the input, often six orders of magnitude from last layer to first. Exploding shows spiking norms and loss spikes, usually on a few steps. Clipping handles the spikes; ReLU, residual connections, and correct initialization handle the decay.
Interview Questions
Q1: Walk through the backward recurrence for one layer.
Start from delta(l) = dL/dz(l). Multiply by the activation derivative elementwise to get dL/dh(l) at the previous level, outer product with the saved input h(l-1) to get dL/dW(l), and transpose W(l) to route error down. Three cheap tensor ops per layer, all reusing values saved on the forward pass.
Q2: Why does reverse mode scale to 100 million parameters but forward mode does not?
Reverse mode gives the entire gradient in roughly two forward costs because the scalar loss collapses the output dimension to one. Forward mode differentiates one input at a time, so 100 million parameters would mean 100 million sweeps, which no budget supports.
Q3: You see gradient norms of 1e-2, 1e-5, 1e-9, 1e-13 across four layers. What is wrong and what fixes it?
That is geometric decay, the vanishing gradient signature, most often sigmoid activations (max derivative 0.25) compounded down the chain. Replace with ReLU or GELU, add residual skip connections to shorten the chain, and initialize so each layer's spectral norm starts near 1.