Sequence Models
Sequence-to-Sequence — Translating Input Sequences to Output Sequences
Seq2seq models are the backbone of machine translation, summarization, and dialogue systems. They encode variable-length inputs into a fixed representation and decode it into variable-length outputs, enabling end-to-end learning of sequence transduction.
- Key point 1 — Encoder-decoder architecture maps variable-length input to variable-length output
- Key point 2 — Teacher forcing stabilizes training but introduces exposure bias
- Key point 3 — Beam search and attention mechanisms unlock high-quality generation
"To translate is to understand — and seq2seq models learn to understand language itself."
Sequence-to-Sequence Models
What You'll Learn
- 1Write the encoder, decoder and output distribution equations of a seq2seq model
- 2Explain the information bottleneck in the fixed-length context vector
- 3Trade teacher forcing against exposure bias and scheduled sampling
- 4Score greedy decoding and length-normalised beam search for generation
Seq2seq models map an input sequence to an output sequence of potentially different length. They are the foundation of machine translation, summarization, and dialogue systems.
Encoder-Decoder Architecture
Factorising the joint probability over the target is what makes autoregressive decoding legal: every step only needs the prefix, so the same weights can generate one token at a time at inference without ever seeing the future.
Context Vector
As the source sentence grows the encoder has to drop detail to keep c at a fixed size, and the decoder has no way to ask for it back. Removing this bottleneck is exactly what attention does, as the next lesson covers.
Teacher Forcing
Each teacher-forced step therefore sees the correct prefix, and the sequence loss splits into independent classification problems that train quickly and with low variance. The price is exposure bias: at inference the decoder feeds its own output, and a single early mistake puts it in a prefix it has never seen during training.
Decoding Strategies
Greedy Decoding
Greedy decoding costs one forward pass per target token, so it is O(T') in time and never stores more than one hypothesis. It is also not the argmax of the full sequence: a token that ranks second today can lead to a much better suffix later, and greedy has no way to revisit that choice.
Beam Search
Without the normaliser the sum of log probabilities is always more negative for a longer output, so beam search would end every sentence as early as the model allows. With near 0.6 the criterion behaves like a length averaged probability, and translation quality improves measurably over the raw sum.
Complete PyTorch Implementation
Training
Practice Exercises
-
Implement beam search: Write a beam search decoder with length normalization. Compare outputs with greedy decoding.
-
Scheduled sampling: Modify the training loop to use scheduled sampling with linear decay of from 1.0 to 0.0.
-
Transformer decoder: Replace the RNN decoder with a Transformer decoder. Compare performance and training speed.
-
Summarization task: Train a seq2seq model on CNN/DailyMail dataset for text summarization. Evaluate with ROUGE scores.
Summary
What to Learn Next
-> Attention Mechanisms Discover how attention solves the information bottleneck in sequence models.
-> LSTM Networks Explore gated recurrent units with cell state for long-range dependencies.
-> GRU Networks Learn a simpler gating mechanism with fewer parameters and comparable performance.
-> RNN Deep Dive Understand recurrent fundamentals and the vanishing gradient problem.
-> Vision Transformers Apply Transformer architecture to image recognition by treating patches as tokens.
-> DL Systems Design Master distributed training, monitoring, and production deployment of deep learning models.
Common Mistakes
Treating the context vector as sufficient for long inputs
Fix: Past roughly 30 tokens one vector of fixed width cannot hold the source; switch to an attention based decoder that can revisit every encoder state.
Training with teacher forcing and reporting greedy BLEU
Fix: Decay the forcing probability during training or at least report both numbers, otherwise the metric measures a decoding mode the model never adapted to.
Ranking beams by the raw sum of log probabilities
Fix: The sum is always more negative for a longer output, so divide by the length raised to alpha between 0.6 and 1.0 before keeping the top k.
Best Practices
- Warm up the learning rate over the first few thousand steps and clip gradient norm at 1.0, the loss curve is spiky otherwise
- Apply label smoothing of 0.1 so the decoder does not collapse onto a handful of high frequency tokens
- Validate with greedy decoding throughout training and reserve beam search with alpha 0.6 for the final number
Key Takeaways
- The decoder models P of y given x as a product over positions, so generation is one autoregressive step at a time
- A single context vector must carry the whole source, which is exactly the bottleneck attention removes
- Teacher forcing stabilises training and parallelises the loss but leaves the model inexperienced with its own outputs
- Beam search with length normalisation beats greedy; without the normaliser it prefers the shortest possible sentence
Frequently Asked Questions
What exactly is exposure bias?
It is the mismatch between training and inference. During training the decoder always receives the gold token as its previous input, so it never learns to recover from its own mistake. At inference it feeds its own predictions, and once it slips off the reference path it lands in states it has never seen. Scheduled sampling exists to close that gap, and beam search exists to limit the damage when it happens.
Why does beam search need length normalisation?
Log probabilities are negative, so appending another token always lowers the raw sum. A candidate scored on the raw sum therefore wins by being short, and a decoder left to itself will happily emit one confident word. Dividing by length raised to a power alpha between 0.6 and 1.0 rebalances the criterion so a longer candidate can win on average probability rather than on brevity.
Does seq2seq still matter now that Transformers won?
The pattern outlived the RNN. Any task that maps one variable length sequence to another, translation, transcription, summarisation, speech to text, still uses an encoder-decoder skeleton with an autoregressive decoder and usually beam search at inference. What changed is that both sides became stacks of self attention rather than recurrent cells.
Interview Questions
Q1: Write the seq2seq training objective.
The negative log likelihood of the reference given the source, L equals minus the sum over t of log P of y_t star given the gold prefix y_<t star and x. Teacher forcing makes every term computable in one parallel pass because the prefix is always the gold one.
Q2: Why does a fixed context vector hurt on long inputs?
The encoder must map a growing number of hidden states onto a vector of fixed width n, so detail is discarded as the source grows. The decoder can query only that one vector, so a dropped detail is unrecoverable and BLEU falls off sharply past a few dozen tokens.
Q3: How do you debug a model with good teacher-forced loss but poor BLEU?
Split the gap. Run greedy decoding on validation and compare with perplexity, then try beam search with and without length normalisation. If greedy is fine and beam is worse the normaliser is wrong; if both are bad while the loss is good it is exposure bias, and scheduled sampling is the next lever.