DL Foundations
Weight Initialization — The Hidden Key to Training Deep Networks
Poor weight initialization causes vanishing or exploding gradients, making deep networks untrainable before training even begins. Proper initialization preserves activation variance across layers.
- Xavier for Sigmoid/Tanh — Preserves variance for activations that are approximately linear near zero
- He/Kaiming for ReLU — Doubles variance to compensate for ReLU zeroing out half the activations
- ResNet Zero-Init Trick — Initialize last batch norm to zero in residual blocks for stable deep training
Weight Initialization — Xavier, He, LSUV and Variance Preservation
Weight initialization determines how neural network parameters are set before training begins. Poor initialization leads to vanishing or exploding gradients, making deep networks untrainable.
Why Initialization Matters
Random Initialization
Xavier/Glorot Initialization
He/Kaiming Initialization
Orthogonal Initialization
LSUV (Layer-Sequential Unit-Variance)
Initialization for ResNets
Practical Recommendations
Summary
- Xavier/Glorot for sigmoid/tanh: preserves variance
- He/Kaiming for ReLU: doubles variance to compensate for ReLU
- Orthogonal for RNNs: preserves gradient magnitude through time
- Zero-init for ResNet last BN: ensures identity mapping at initialization
- LSUV: data-driven, works for any architecture