Transformers
Vision Transformers — Applying Transformers to Images
Vision Transformers bring the power of attention to computer vision by treating image patches as tokens. With patch embedding, positional encoding, and self-attention, ViT captures global relationships from the first layer — outperforming CNNs when trained on sufficient data.
- Key point 1 — Patch embedding converts images into sequences of flattened patches
- Key point 2 — No inductive bias means ViT needs large datasets to learn locality
- Key point 3 — DeiT and Swin Transformer solve data efficiency and scalability
"When images become sequences, attention becomes vision."
Vision Transformers — ViT and Beyond
What You'll Learn
- 1Derive the patch count, token sequence and embedding shapes for a given image size
- 2Explain why ViT needs large pretraining data where a CNN does not
- 3Compare the parameter budget of ViT-B/16 against a ResNet-50
- 4Select patch size and attention window for a target resolution
Vision Transformers (ViT) apply the Transformer architecture to image recognition by treating image patches as tokens, achieving competitive or superior performance to CNNs.
From NLP to Vision
Patch Embedding
A 224 square image cut into 16 pixel patches becomes a sequence of 196 tokens plus a class token, so the encoder sees 197 positions. Halving the patch size to 8 quadruples the sequence to 784 tokens and therefore multiplies the attention cost by sixteen, which is the single most important budget knob in ViT.
Positional Encoding
ViT Architecture
Each encoder block spends weights on the attention projections and on the two feedforward layers of width , so the budget depends only on depth and width. ViT-B/16 therefore lands at roughly 86 million parameters including the patch projection and the classifier, comparable to a ResNet-50 with a very different shape.
How Vision Transformers process images: This diagram shows the complete ViT pipeline. A 224×224 input image is first split into 196 patches of 16×16 pixels each (14×14 grid). The Patch Embedding layer (green) is actually a Conv2d with kernel=16 and stride=16 — it cuts the image into patches and projects each to D=768 dimensions in one operation. A special [CLS] token (yellow) is prepended to the sequence — this token's final representation is used for classification. Positional embeddings (purple) are added to give spatial information (since patches are unordered). The sequence enters the Transformer Encoder (pink) with L=12 layers of multi-head self-attention + feed-forward networks. The key advantage over CNNs: self-attention captures global relationships from layer 1 — a patch in the top-left can directly attend to a patch in the bottom-right, without needing to stack many convolutional layers. The Classification Head (red) maps the CLS token output to 1000 class probabilities.
ViT vs CNN
| Aspect | CNN | ViT |
|---|---|---|
| Inductive bias | Local connectivity, translation equivariance | None (global attention) |
| Data requirement | Works with less data | Needs large dataset (JFT-300M) |
| Computational cost | per layer | self-attention |
| Patch size | N/A | Determines sequence length |
| Position info | Built-in (conv) | Explicit positional encoding |
| Transfer learning | Excellent | Excellent with pretraining |
With and the crossover sits at patches, so at the usual 196 patches attention is far cheaper than a convolution of the same width, million against billion multiply-accumulates. What ViT pays for is statistical, not arithmetic: without local connectivity it must learn locality from data.
DeiT (Data-efficient Image Transformers)
Swin Transformer
PyTorch Implementation
Practice Exercises
-
Patch visualization: Split an image into patches and visualize the embedding space using t-SNE.
-
Position embedding analysis: Train a ViT and visualize learned positional embeddings. Do they capture spatial structure?
-
Swin Transformer: Implement shifted window attention. Verify linear complexity on large images.
-
Data augmentation: Compare ViT performance with and without DeiT-style augmentation on ImageNet-1K.
Key Takeaways
What to Learn Next
-> Attention Mechanisms Discover how attention solves the information bottleneck in sequence models.
-> CNN Architecture Deep Dive Master convolutional layers, pooling, and modern CNN architectures.
-> Object Detection Learn to detect and localize objects in images with YOLO and Faster R-CNN.
-> Semantic Segmentation Classify every pixel in an image for detailed scene understanding.
-> DL Systems Design Master distributed training, monitoring, and production deployment of deep learning models.
-> Model Compression Make deep learning models fast and efficient for production deployment.