Sequence Models
RNNs Deep Dive — Understanding Recurrent Neural Networks
Recurrent Neural Networks process sequential data by maintaining a hidden state that carries information across time steps. This tutorial covers vanilla RNNs and their fundamental limitations.
- BPTT Unrolls Through Time — Backpropagation through time treats the RNN as a deep feedforward network
- Vanishing Gradients Limit Memory — The product of Jacobians shrinks exponentially, limiting to ~10-20 steps
- Orthogonal Initialization Helps — Setting recurrent weights to orthogonal matrices preserves gradient magnitude
RNN Deep Dive — Vanilla RNN, BPTT and Vanishing/Exploding Gradients
What You'll Learn
- 1Write the vanilla RNN recurrence and count the parameters it shares across time
- 2Derive backpropagation through time by unrolling the recurrence into a computational graph
- 3Explain vanishing and exploding gradients as powers of the recurrent weight matrix
- 4Apply clipping, orthogonal initialization and gating to train stable sequence models
Recurrent Neural Networks process sequential data by maintaining a hidden state that carries information across time steps. This tutorial covers vanilla RNNs and their fundamental limitations. The central fact to hold on to is that time steps are layers with tied weights, so every gradient rule you know from deep feedforward nets applies here, multiplied times over.
Why Recurrence?
Vanilla RNN
Weight sharing keeps the parameter count independent of sequence length. With hidden size and input size the recurrent and input blocks alone hold numbers, reused at every one of the steps:
Backpropagation Through Time (BPTT)
The sum runs over all steps but each term carries a different number of factors: the gradient from step 1 multiplies roughly times, while the gradient from step multiplies it once. Storing the hidden states for that backward pass is why BPTT memory grows linearly with sequence length, and why truncated BPTT cuts the graph after a fixed window, typically 100 steps, while still carrying the hidden state forward.
Vanishing and Exploding Gradients
An error signal travelling 99 steps back is then 0.003 percent of its original size, and after the optimizer normalizes the update the useful component sits below float 32 noise. The sign of the exponent is the whole story: a spectral norm of 0.95 kills long range credit assignment inside 200 steps, while 1.05 blows the same gradient up past 1e20.
Solutions to Vanishing Gradients
Clipping rescales instead of truncating, so the update keeps its direction: a batch whose gradient norm reaches 50 under is divided by 50, which is the difference between one bad step and a wiped model. Clip on the global norm across all parameters, not per parameter, or a single large matrix hides the rest.
Applications
Practical Considerations
Summary
- Vanilla RNN maintains a hidden state that evolves over time through recurrence
- BPTT unrolls the RNN and applies backpropagation through the unrolled graph
- Vanishing gradients limit vanilla RNNs to ~10-20 time steps
- Exploding gradients can be mitigated with gradient clipping
- Solutions: LSTM/GRU (gating), orthogonal initialization, skip connections
- Transformers have largely replaced RNNs for most sequence tasks due to parallelization
Next: LSTM Networks
Common Mistakes
Training an RNN with no gradient clipping
Fix: Clip the global norm at 1.0 or 5.0: one 40 step sequence can otherwise emit a 1e20 gradient and destroy the weights.
Feeding the model its own predictions from step one
Fix: Use teacher forcing with the ground truth token as the next input, then decay the ratio with scheduled sampling.
Expecting the parameter count to grow with sequence length
Fix: Weights are tied across time, so P depends only on n and d while compute grows linearly with T.
Best Practices
- Initialize W_hh as an orthogonal matrix so its singular values sit at 1 and the gradient neither explodes nor vanishes
- Pack padded sequences before the loop so the final hidden state comes from a real token rather than a pad
- Log the gradient norm every step: a curve that climbs then collapses is the signature of exploding gradients
Key Takeaways
- The recurrence h_t = tanh(W_hh h_{t-1} + W_xh x_t + b) reuses one set of weights at all T steps
- BPTT unrolls the graph and backpropagates through a product of Jacobians that each contain W_hh
- Gradient size scales as the Tth power of the spectral norm of W_hh: 0.9 to the 99th power is 2.6e-5
- Clipping, orthogonal initialization and gating with LSTM or GRU are the three standard fixes
Frequently Asked Questions
Why does an RNN behave like a very deep network?
Each time step applies the same weight matrices one more time, so a 100 step sequence is a 100 layer network with tied weights. Backpropagation multiplies 100 Jacobians together, and it is that chain of multiplications, not the architecture on paper, that makes the gradient at the first step disappear.
How long can a vanilla RNN actually remember?
With a spectral norm of 0.9 the gradient reaching step 1 after 99 steps is 0.9 to the 99th power, about 2.6e-5 of its original value, so useful credit assignment stops at roughly 10 to 20 steps. On the classic adding problem vanilla RNNs fail beyond about 20 terms while an LSTM solves sequences of several hundred.
Does truncated BPTT throw away the history?
No. It cuts the gradient after a window of k steps, usually 100, but the hidden state still flows forward through the whole sequence, so the model sees the full history while credit assignment only travels k steps back. Memory drops from O(T) to O(k), and long range dependencies must be handled by the state itself rather than by the gradient.
Interview Questions
Q1: Write the gradient of the loss with respect to W_hh and explain its structure.
It is a sum over time steps, and every term contains a product of Jacobians diag(sigma'(z_j)) W_hh, so the term from step 1 carries W_hh about T times while the term from step T carries it once. After 50 steps the earliest terms hold W_hh to the 49th power.
Q2: Why is a vanishing gradient a learning problem rather than a numerics problem?
The gradient is the learning signal. When it falls to 1e-5 the honest component of the update is smaller than float32 rounding after normalization, so the network never learns that an input forty steps earlier affected today's output.
Q3: What does gradient clipping fix, and what does it leave broken?
It bounds the step when a rare sequence produces an enormous norm, so one batch cannot wipe out learned weights. It does nothing for vanishing gradients, which are too small rather than too large, so gating or orthogonal initialization is still required.