Prerequisites

Before diving into RNNs, you should be comfortable with:

• Neural Network Fundamentals: Forward/backward pass, activation functions, backpropagation (see Tutorial 21)
• Sequence Data: Time series, text as token sequences, variable-length inputs
• PyTorch Basics: Tensors, nn.Module, embedding layers, padding/packing sequences
• Linear Algebra: Matrix multiplication, recurrent weight matrices

Learning Objectives

By the end of this tutorial, you will be able to:

1. Explain how vanilla RNNs process sequential data and their limitations
2. Derive the LSTM gate equations and understand why they solve vanishing gradients
3. Compare GRU and LSTM architectures and choose between them
4. Implement bidirectional RNNs for full-context processing
5. Build encoder-decoder (seq2seq) architectures for translation tasks
6. Implement LSTM models in PyTorch for classification and generation
7. Identify when to use RNNs vs Transformers for sequential tasks

RNN, LSTM and GRU — Complete Guide

Recurrent networks process sequential data by maintaining a hidden state that carries information across time steps. Unlike Transformers, they process one token at a time with memory per step but sequential operations for a sequence of length .

Vanilla RNN

At each time step , the RNN computes:

LSTM (Long Short-Term Memory)

LSTM (Hochreiter and Schmidhuber, 1997) introduces a cell state as an information highway, with three gates controlling information flow:

GRU (Gated Recurrent Unit)

GRU (Cho et al., 2014) simplifies LSTM by merging the cell and hidden state and using only two gates:

Bidirectional RNN

Processes the sequence in both directions and concatenates hidden states:

Use case: NER, sentiment analysis — where full context is available. Cannot be used for autoregressive generation.

PyTorch Implementation

Real-World Applications

Common Mistakes & How to Avoid Them

Key Formulas Reference

Interview Questions

Practice Exercise

import torch
import torch.nn as nn

# Task 1: Implement an LSTM-based sentiment classifier
# - Embedding layer: vocab_size -> 128
# - LSTM: 128 -> 256, 2 layers, bidirectional
# - Attention mechanism over LSTM outputs
# - Classifier: 512 -> 128 -> 2 (positive/negative)

class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size, embed_dim=128, hidden_dim=256, num_classes=2):
        super().__init__()
        # Your implementation here
        pass

    def forward(self, x, lengths):
        # Your implementation here
        # Remember to pack_padded_sequence!
        pass

# Task 2: Train on IMDB dataset
# - Use pretrained GloVe embeddings (optional)
# - Adam optimizer, lr=1e-3
# - Gradient clipping at 5.0
# - Achieve >85% accuracy

# Task 3: Implement simple attention over LSTM outputs
# attention_weight = torch.softmax(torch.bmm(lstm_out, query), dim=1)
# context = torch.bmm(attention_weight.permute(0,2,1), lstm_out)

Key Takeaways

What to Learn Next

Advanced Topics

Attention Mechanism for RNNs

The attention mechanism was originally designed to solve the bottleneck problem in Seq2Seq models:

The decoder receives a context vector that is a weighted sum of all encoder hidden states, where the weights indicate which input tokens are most relevant at decoding step .

Truncated Backpropagation Through Time (TBPTT)

Full BPTT unrolls the entire sequence, which is memory-intensive for long sequences. TBPTT processes the sequence in fixed-length chunks:

1. Forward pass for steps
2. Backward pass for those steps
3. Detach the hidden state from the computation graph
4. Continue with the next chunk

Key parameter: chunk size — determines how far back gradients flow. Common values: 35-100 steps.

Weighted softmax in Attention

Standard softmax computes . For numerical stability and better gradients:

This shift doesn't change the result but prevents overflow in floating-point computation.

Stacking and Deep RNNs

Deep RNNs stack multiple recurrent layers:

Recommendations:

• 2-3 layers is optimal for most tasks
• Add dropout between layers (not within time steps)
• Use skip connections from layer to layer for deeper RNNs
• Layer norm helps stabilize deep recurrent training

Comparison Table

Further Reading

• Hochreiter, S. & Schmidhuber, J. (1997). "Long Short-Term Memory." — The original LSTM paper, a landmark in deep learning.
• Cho, K. et al. (2014). "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." — The GRU paper.
• Sutskever, I. et al. (2014). "Sequence to Sequence Learning with Neural Networks." — Foundational seq2seq paper.
• Bahdanau, D. et al. (2014). "Neural Machine Translation by Jointly Learning to Align and Translate." — Attention mechanism for seq2seq.
• Pascanu, R. et al. (2013). "On the difficulty of training Recurrent Neural Networks." — Gradient clipping and vanishing gradient analysis.
• Colah's Blog: Understanding LSTM Networks — The most famous visual explanation of LSTM internals.

Quick Reference Cheat Sheet