Sequence Models
RNNs Deep Dive — Understanding Recurrent Neural Networks
Recurrent Neural Networks process sequential data by maintaining a hidden state that carries information across time steps. This tutorial covers vanilla RNNs and their fundamental limitations.
- BPTT Unrolls Through Time — Backpropagation through time treats the RNN as a deep feedforward network
- Vanishing Gradients Limit Memory — The product of Jacobians shrinks exponentially, limiting to ~10-20 steps
- Orthogonal Initialization Helps — Setting recurrent weights to orthogonal matrices preserves gradient magnitude
RNN Deep Dive — Vanilla RNN, BPTT and Vanishing/Exploding Gradients
Recurrent Neural Networks process sequential data by maintaining a hidden state that carries information across time steps. This tutorial covers vanilla RNNs and their fundamental limitations.
Why Recurrence?
Vanilla RNN
Backpropagation Through Time (BPTT)
Vanishing and Exploding Gradients
Solutions to Vanishing Gradients
Applications
Practical Considerations
Summary
- Vanilla RNN maintains a hidden state that evolves over time through recurrence
- BPTT unrolls the RNN and applies backpropagation through the unrolled graph
- Vanishing gradients limit vanilla RNNs to ~10-20 time steps
- Exploding gradients can be mitigated with gradient clipping
- Solutions: LSTM/GRU (gating), orthogonal initialization, skip connections
- Transformers have largely replaced RNNs for most sequence tasks due to parallelization
Next: LSTM Networks