DL Foundations
Optimizers — From SGD to Adam, Finding the Best Weights
Optimizers determine how neural network parameters are updated based on computed gradients. The choice of optimizer and learning rate schedule significantly impacts training speed and final performance.
- AdamW is the Default — Combines momentum with adaptive learning rates and decoupled weight decay
- SGD + Momentum for Vision — Often achieves better generalization than adaptive methods on image tasks
- Cosine Annealing + Warmup — The standard learning rate schedule for modern deep learning
Optimizers for Deep Learning — SGD, Adam, AdamW and Learning Rate Schedules
Optimizers determine how neural network parameters are updated based on computed gradients. The choice of optimizer and learning rate schedule significantly impacts training speed and final performance.
Gradient Descent Foundation
SGD with Momentum
Adaptive Learning Rate Methods
AdaGrad
RMSProp
Adam and AdamW
Learning Rate Schedules
Step Decay
Cosine Annealing
Warmup
Optimizer Selection Guide
Practical Tips
Summary
- SGD + Momentum for computer vision: best generalization with proper tuning
- AdamW for NLP/transformers: fast convergence, decoupled weight decay
- Cosine annealing + warmup is the standard learning rate schedule
- Learning rate is the most important hyperparameter — tune it first
- Different tasks require different optimizers and hyperparameters
Next: Weight Initialization