DL Foundations
Optimizers - From SGD to Adam, Finding the Best Weights
Optimizers determine how neural network parameters are updated based on computed gradients. The choice of optimizer and learning rate schedule significantly impacts training speed and final performance.
- AdamW is the Default - Combines momentum with adaptive learning rates and decoupled weight decay
- SGD + Momentum for Vision - Often achieves better generalization than adaptive methods on image tasks
- Cosine Annealing + Warmup - The standard learning rate schedule for modern deep learning
Optimizers for Deep Learning - SGD, Adam, AdamW and Learning Rate Schedules
What You'll Learn
- 1Derive momentum, AdaGrad, RMSProp, and Adam updates from the plain SGD step
- 2Explain bias correction and why Adam needs it in the first few steps
- 3Contrast L2 regularization inside Adam with decoupled weight decay in AdamW
- 4Combine warmup with cosine annealing and pick a starting learning rate
Optimizers determine how neural network parameters are updated based on computed gradients. The choice of optimizer and learning rate schedule significantly impacts training speed and final performance.
Gradient Descent Foundation
Mini-batch gradients are unbiased estimates of the full gradient, and their noise is the point: stochastic noise helps escape shallow minima. The variance of that noise scales as for batch size , which is why doubling the batch lets you roughly double the learning rate.
SGD with Momentum
Unrolling the recurrence shows momentum is just an exponential moving average of past gradients, and the fraction of history retained after steps is :
With the window holds about 65 percent of its mass after 10 steps and 99.8 percent after 60, so momentum both averages away per-step noise and keeps pushing in directions where gradients persist for dozens of steps. This is why the momentum path in the diagram crosses the valley walls smoothly while plain SGD zigzags across them.
Adaptive Learning Rate Methods
AdaGrad
RMSProp
Adam and AdamW
The EMA starts at zero, so early estimates are biased toward zero; dividing by removes that bias exactly. The correction factor is huge at first and gone within a few hundred steps:
Without this, the first updates would be roughly ten times too small and warmup would be partly redundant. Note also that dividing by makes the typical step magnitude per parameter about regardless of gradient scale, which is why Adam tolerates a fixed learning rate across very different architectures.
Learning Rate Schedules
Step Decay
Cosine Annealing
Warmup
Optimizer Selection Guide
Practical Tips
Summary
- SGD + Momentum for computer vision: best generalization with proper tuning
- AdamW for NLP/transformers: fast convergence, decoupled weight decay
- Cosine annealing + warmup is the standard learning rate schedule
- Learning rate is the most important hyperparameter - tune it first
- Different tasks require different optimizers and hyperparameters
Next: Weight Initialization
Common Mistakes
Running Adam at lr 1e-3 on a vision model where SGD at lr 0.1 is the baseline
Fix: Match the optimizer to the domain: AdamW around 1e-4 to 3e-4 with warmup for transformers, SGD+momentum 0.1 with cosine for ResNets
Adding weight decay via the loss and still calling it Adam
Fix: The L2 term gets divided by sqrt(v_hat), so decay silently shrinks; use AdamW and set weight_decay directly on the optimizer
Skipping warmup because the schedule looks fine in a short run
Fix: At large batch or large model scale the first few hundred steps have unreliable v_hat and normalizer statistics; keep at least 1 to 2 percent of steps as warmup
Best Practices
- Tune the learning rate first with a range test, then fix the optimizer and tune weight decay
- Log the learning rate per step so a schedule bug cannot masquerade as a model bug
- When comparing optimizers, hold the effective batch and total steps constant or the comparison is meaningless
Key Takeaways
- Momentum is an exponential moving average of gradients: 1 - beta^k of the mass arrives after k steps
- AdaGrad's accumulator grows without bound so its rate decays as 1/sqrt(t); RMSProp and Adam replace it with an EMA
- Adam divides by sqrt(v_hat) so typical per-parameter steps are about eta, and bias correction inflates the first steps by up to 1/(1 - beta1)
- AdamW applies decay as (1 - eta lambda) times the weight, decoupled from the adaptive scaling that neutralizes L2
Frequently Asked Questions
Why does warmup matter if cosine annealing already starts at the peak?
Cosine starts at immediately, but the first steps run with estimated from a handful of samples and with normalizer statistics that have not settled. A peak-rate step under those conditions can spike the loss permanently; warmup trades a few hundred cheap steps for a stable estimate of the second moment before full rate is applied.
Is SGD with momentum really better than Adam for vision?
On classic ResNet training with carefully tuned schedules, tuned SGD+momentum often lands a point or two better in final accuracy and generalizes more robustly to distribution shift. Adam reaches a given training loss in far fewer steps and needs less tuning, which is why it dominates transformers and why the choice is a budget question as much as an accuracy question.
What happens if I set beta2 too low or too high?
Too low (say 0.99) makes track recent gradient spikes, so the effective learning rate jumps around and training can stall in noisy regions; too high (0.9999) makes the normalizer adapt slowly, so the first few thousand steps are governed by bias correction alone and large transient gradients pass through nearly unscaled. The 0.999 default is the sweet spot for step counts in the hundreds of thousands.
Interview Questions
Q1: Derive the Adam update from momentum and RMSProp.
Take the EMA of gradients m = beta1 m + (1-beta1) g and of squared gradients v = beta2 v + (1-beta2) g^2, bias correct both by dividing by 1 - beta^t, then divide the normalized direction by sqrt(v_hat) plus epsilon: theta -= eta m_hat / (sqrt(v_hat) + eps). Momentum supplies the direction, RMSProp supplies the per-parameter scale.
Q2: Why does weight decay behave differently in Adam than in SGD?
In SGD, L2 and weight decay are algebraically identical because the update is linear in the gradient. In Adam the update is divided by sqrt(v_hat), and the L2 contribution also inflates v_hat, so the decay each weight receives depends on its own history. AdamW applies theta -= eta * lambda * theta outside the adaptive division, restoring the fixed shrinkage SGD gives you.
Q3: You are training a transformer and loss diverges in the first 500 steps. What do you check first?
The learning rate trace: confirm warmup is actually ramping rather than jumping to the peak, check that gradients are not spiking before v_hat has filled in, verify beta2 and epsilon defaults, and rule out a missing or mis-scaled loss reduction before touching anything else.