Prerequisites

Before diving into Transformers, you should be comfortable with:

• Neural Network Fundamentals: Forward/backward pass, activation functions, backpropagation (see Tutorial 21)
• RNNs and LSTMs: Sequential processing, vanishing gradients, attention motivation (see Tutorial 23)
• Linear Algebra: Matrix multiplication, softmax, dot-product similarity
• PyTorch: nn.Module, batch processing, mask operations
• NLP Basics: Tokenization, embeddings, sequence-to-sequence tasks

Learning Objectives

By the end of this tutorial, you will be able to:

1. Implement scaled dot-product self-attention from scratch
2. Explain why multi-head attention captures different relationship types
3. Compare sinusoidal, learned, and rotary positional encodings
4. Differentiate encoder (BERT) vs decoder (GPT) architectures
5. Understand the quadratic complexity bottleneck and its solutions
6. Build a Transformer block in PyTorch with proper masking
7. Explain why Transformers replaced RNNs for most NLP tasks

Transformers — Attention Is All You Need

Transformers (Vaswani et al., 2017) replaced RNNs as the dominant architecture for sequence processing. They achieve sequential operations (vs for RNNs), enabling massive parallelism on GPUs.

Self-Attention

The core mechanism: each token attends to every other token to compute a weighted representation.

Multi-Head Attention

Multiple attention heads capture different types of relationships simultaneously:

where

Positional Encoding

Since self-attention is permutation-invariant, we must inject position information:

Transformer Block

Encoder vs Decoder

• Encoder (BERT): Bidirectional self-attention — every token attends to every other token. Good for understanding tasks (classification, NER, QA).
• Decoder (GPT): Masked self-attention — each token attends only to previous tokens. Good for text generation, completion.

PyTorch Implementation

Real-World Applications

Common Mistakes & How to Avoid Them

Key Formulas Reference

Interview Questions

Practice Exercise

import torch
import torch.nn as nn
import math

# Task 1: Implement scaled dot-product attention
def scaled_dot_product_attention(Q, K, V, mask=None):
    # Compute attention scores
    # Apply mask if provided
    # Apply softmax and scale
    # Return weighted sum of values
    pass

# Task 2: Implement multi-head attention
class MultiHeadAttention(nn.Module):
    def __init__(self, d_model=512, n_heads=8):
        super().__init__()
        # Create Q, K, V projection layers
        # Create output projection layer
        pass

    def forward(self, Q, K, V, mask=None):
        # Split into heads
        # Apply attention per head
        # Concatenate and project
        pass

# Task 3: Implement positional encoding
class SinusoidalPositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        # Create sinusoidal encoding matrix
        pass

    def forward(self, x):
        # Add positional encoding to input
        pass

# Task 4: Assemble full Transformer encoder
class TransformerEncoder(nn.Module):
    def __init__(self, d_model=512, n_heads=8, n_layers=6):
        super().__init__()
        # Stack of Transformer blocks
        pass

# Task 5: Train on a language modeling task
# - Use WikiText-2 dataset
# - Autoregressive (next token prediction)
# - Adam optimizer with warmup schedule

Key Takeaways

What to Learn Next

Advanced Topics

Efficient Attention Mechanisms

Standard self-attention has O(n^2) complexity. Several methods address this:

• Sparse Attention (Longformer, BigBird): Attend to local windows (e.g., 512 tokens) + a few global tokens. Complexity: O(n * window_size).
• Linear Attention (Performer): Approximate softmax with kernel functions to avoid materializing the full n x n matrix.
• Flash Attention (Dao et al.): Compute attention in tiles on GPU SRAM, using O(n) memory instead of O(n^2). 2-4x speedup with zero approximation.
• Sliding Window (Mistral): Each token attends only to a fixed window of neighbors, reducing complexity to O(n * w).

Scaling Laws

Kaplan et al. (2020) discovered power law relationships:

where is loss, is parameters, is data tokens, is compute FLOPs.

Key insight: Optimal performance requires balancing all three. Doubling parameters gives diminishing returns without proportional data/compute increases. This led to Chinchilla's compute-optimal training: for a given budget, train a smaller model on more data.

Transformer Variants

• BERT (Encoder-only): Bidirectional attention, masked language modeling. Good for understanding.
• GPT (Decoder-only): Causal attention, next-token prediction. Good for generation.
• T5 (Encoder-decoder): Text-to-text framework. Any task as text generation.
• Vision Transformer (ViT): Treats image patches as tokens. ViT-L/14 achieves 87.8% on ImageNet.
• Perceiver: Processes arbitrary modalities (text, images, audio, point clouds) through a learned latent space.
• State Space Models (Mamba): Linear-time alternative to Transformers for very long sequences (100K+ tokens).

KV Cache in Inference

During autoregressive generation, previously computed Key and Value matrices are cached to avoid recomputation. Without KV cache, generating tokens requires total computation. With KV cache, each new token requires only computation (for the new token's Q against all cached KVs).

Comparison Table

Further Reading

• Vaswani, A. et al. (2017). "Attention Is All You Need." — The original Transformer paper, one of the most influential in ML history.
• Devlin, J. et al. (2018). "BERT: Pre-training of Deep Bidirectional Transformers." — Bidirectional encoder for language understanding.
• Radford, A. et al. (2018/2019). "Improving Language Understanding by Generative Pre-Training." — GPT-1 and GPT-2.
• Kaplan, J. et al. (2020). "Scaling Laws for Neural Language Models." — Power law relationships between compute, data, and performance.
• Dao, T. et al. (2022). "FlashAttention: Fast and Memory-Efficient Exact Attention." — Hardware-aware attention algorithm.
• Su, J. et al. (2021). "RoFormer: Enhanced Transformer with Rotary Position Embedding." — RoPE for long-context models.

Quick Reference Cheat Sheet

Model Comparison: Encoder vs Decoder vs Encoder-Decoder