DL Foundations
Regularization in Deep Learning — Dropout, Batch Norm, and More
Deep networks have far more parameters than training samples, making them prone to overfitting. Regularization techniques constrain the model to improve generalization to unseen data.
- Dropout as Ensemble — Randomly zeroing neurons is equivalent to training an ensemble of subnetworks
- BatchNorm Stabilizes — Normalizes per-batch activations, allowing 10-100x higher learning rates
- Combine Techniques — Different regularization methods are complementary; the key is avoiding over-regularization
Regularization for Deep Learning — Dropout, BatchNorm, Data Augmentation and Weight Decay
Deep networks have far more parameters than training samples, making them prone to overfitting. Regularization techniques constrain the model to improve generalization.
The Overfitting Problem
Dropout
Batch Normalization
Layer Normalization
Weight Decay
Data Augmentation
Early Stopping
Regularization Summary
Summary
- Dropout: Randomly zeros neurons during training, acts as implicit ensemble
- BatchNorm: Normalizes across batch, allows higher learning rates
- LayerNorm: Normalizes across features, preferred for transformers
- Weight Decay: Penalizes large weights, always use with proper value
- Data Augmentation: Increases effective training set size
- Early Stopping: Simple and effective, always monitor validation loss
- Combine multiple regularization techniques for best results