DL Foundations
Weight Initialization - The Hidden Key to Training Deep Networks
Poor weight initialization causes vanishing or exploding gradients, making deep networks untrainable before training even begins. Proper initialization preserves activation variance across layers.
- Xavier for Sigmoid/Tanh - Preserves variance for activations that are approximately linear near zero
- He/Kaiming for ReLU - Doubles variance to compensate for ReLU zeroing out half the activations
- ResNet Zero-Init Trick - Initialize last batch norm to zero in residual blocks for stable deep training
Weight Initialization - Xavier, He, LSUV and Variance Preservation
What You'll Learn
- 1Derive the variance preservation requirement Var(W) = 1/n_in across stacked layers
- 2Explain why Xavier targets sigmoid and tanh while He targets ReLU
- 3Compute the blow-up or collapse factor for a naive init over 10 layers
- 4Choose an initialization scheme per architecture: CNN, transformer, or RNN
Weight initialization determines how neural network parameters are set before training begins. Poor initialization leads to vanishing or exploding gradients, making deep networks untrainable.
Why Initialization Matters
Chaining the recurrence from the input to layer turns the per-layer condition into a product, and every factor off by a little is multiplied rather than added:
A 5 percent error per layer compounds to after 50 layers. This is why initialization is a property of the distribution you draw from, not a property you can fix with a single lucky seed.
Random Initialization
Xavier/Glorot Initialization
Xavier is not pulled from thin air: preserving variance forward wants , preserving gradient variance backward wants , and the standard solution satisfies both as well as possible by taking the geometric mean:
The uniform form uses as its half-width because a uniform on has variance , giving the same second moment as the Gaussian.
He/Kaiming Initialization
The derivation is one line once you know ReLU zeroes half the mass of a symmetric distribution. If then has variance , so you must double the variance you inject to land back at times equal to the input variance:
With Xavier instead, each ReLU layer multiplies variance by : after 20 layers activations are times smaller, gradients vanish, and the network never starts learning. This is exactly what the He path in the diagram restores.
Orthogonal Initialization
LSUV (Layer-Sequential Unit-Variance)
Initialization for ResNets
Practical Recommendations
Summary
- Xavier/Glorot for sigmoid/tanh: preserves variance
- He/Kaiming for ReLU: doubles variance to compensate for ReLU
- Orthogonal for RNNs: preserves gradient magnitude through time
- Zero-init for ResNet last BN: ensures identity mapping at initialization
- LSUV: data-driven, works for any architecture
Next: Regularization for Deep Learning
Common Mistakes
Keeping Xavier after switching from tanh to ReLU
Fix: Each ReLU halves variance, so re-init with He or re-check activation variance after the swap; watch the first hundred steps for a loss that never moves
Zero-initializing an entire layer's weights to kill symmetry
Fix: Zero weights give every neuron the same gradient forever; zero only the last batch norm gamma of a residual block, never the weights themselves
Forgetting input-dependent init in transformers
Fix: Embeddings and the output head want a small fixed init around 0.02 while projections want 1/sqrt(d); a single scheme across both miscalibrates at least one of them
Best Practices
- Log activation standard deviations per layer for the first batch as a smoke test of init
- Match the init rule to the activation: He for ReLU family, Xavier for tanh and linear, orthogonal for recurrent weights
- Rely on zero-init batch norm in residual branches when training very deep stacks instead of fighting init per layer
Key Takeaways
- Variance propagates as a product across layers, so a small per-layer bias in Var(W) compounds exponentially
- Xavier is the geometric mean of the forward and backward conditions 1/n_in and 1/n_out, suited to symmetric activations
- He doubles the variance to cancel the half that ReLU clips away, giving 2/n_in
- Orthogonal recurrent weights keep the spectral norm at 1 so gradients neither explode nor vanish through time
- Zero-init of the final batch norm makes each residual block an exact identity with Jacobian I at step zero
Frequently Asked Questions
Why does the factor 2 appear in He but not Xavier?
It compensates for ReLU. Half of a symmetric pre-activation distribution is clipped to zero, so the surviving half carries only of the variance into the next layer; injecting puts that back at parity. Xavier assumes activations pass through roughly unchanged, which holds for tanh near zero but not for the rectifier.
Is initialization still important with batch normalization and residual connections?
Yes, but less so. Normalization re-centers activations every layer and residuals give identity paths, so networks become far more forgiving; however, bad init still shows up as slow early convergence or a spiking loss in the first steps, and transformers without such scaffolding remain sensitive to it. He plus zero-init batch norm remains the safe default for CNN stacks.
How would you verify an initialization in practice?
Run one forward and one backward pass with a random batch before training and print the standard deviation of activations and gradients per layer. With a correct scheme both hover near constant values across depth; a factor-of-two decay per layer or a monotone explosion tells you which rule to swap to.
Interview Questions
Q1: Derive the Xavier variance from the forward and backward requirements.
Forward pass: Var(h_l) = n_in Var(W) Var(h_{l-1}), so Var(W) = 1/n_in preserves signal. Backward pass: Var(g_{l-1}) = n_out Var(W) Var(g_l), so Var(W) = 1/n_out preserves gradient. Satisfying both at once gives the geometric mean sqrt(2/(n_in + n_out)).
Q2: Why do RNNs use orthogonal initialization instead of Xavier?
The recurrent matrix is applied repeatedly, so its spectral norm raised to the power T governs gradient flow through time. Xavier gives a random matrix whose norm is about sqrt(n), leading to exponential blow-up or decay; an orthogonal matrix has all singular values at 1, so the product over T steps stays flat.
Q3: What does setting the last batch norm gamma to zero do to a ResNet block?
The block's contribution becomes zero, so the residual branch drops out entirely at init: the block output equals its input and the Jacobian is the identity. Gradients reach early layers unscaled, and gamma then learns how much of the block to enable during training.