Transfer Learning: Fine-tuning Pre-trained Models
Transfer learning is one of the most powerful paradigms in modern deep learning. Instead of training networks from scratch — a process demanding millions of labeled images and weeks of GPU time — we repurpose knowledge encoded in models trained on massive datasets. This lecture covers the theory, mechanics, and practical strategies for applying transfer learning to real-world problems.
1. What is Transfer Learning?
Transfer learning is the practice of taking a model trained on one task (the source task) and adapting it to a different but related task (the target task).
1.1 Why Transfer Learning Works
Deep networks learn hierarchical representations. Early layers capture universal visual primitives — edges, textures, color gradients. Middle layers compose these into motifs: corners, contours, repeated patterns. Deeper layers encode task-specific semantics — object parts, faces, scenes.
This hierarchy exhibits a key property: lower layers are more general, higher layers are more specific. The same Gabor-like edge detectors useful for ImageNet classification are equally useful for medical image segmentation or satellite imagery analysis.
Formally, consider a source model \mathcal{D}_S = {(x_i^S, y_i^S)}\mathcal{D}_T = {(x_i^T, y_i^T)}\phi_S(x) shares structure with an optimal representation for the target task.
ℹ️
The key insight: natural images share statistical structure. A model that has learned to recognize 1,000 ImageNet classes has implicitly learned useful features for many other visual tasks.
1.2 The Taxonomy of Transfer
| Transfer Type | Source → Target | Example |
|---|---|---|
| Inductive | Same domain, different tasks | ImageNet → X-ray classification |
| Unsupervised | Same domain, no target labels | ImageNet → feature extraction for clustering |
| Domain Adaptation | Different domains, same task | Synthetic → real images for segmentation |
| Multi-task | Shared features, multiple tasks | Single backbone for detection + depth |
1.3 When Does Transfer Help?
The benefit of transfer depends on two factors:
\text{Transfer Gain} \propto \underbrace{\text{Task Similarity}(\mathcal{T}_S, \mathcal{T}T)}{\text{how related are the tasks?}} \times \underbrace{\frac{|\mathcal{D}_S|}{|\mathcal{D}T|}}{\text{data scarcity ratio}}
- High similarity + small target dataset: Maximum benefit (e.g., ImageNet → CIFAR-10)
- Low similarity + large target dataset: Transfer may hurt (negative transfer)
- Any similarity + tiny dataset: Transfer is almost always beneficial
2. Pre-trained Models
2.1 ImageNet and the ILSVRC Benchmark
The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) provided 1.28 million training images across 1,000 categories. This dataset catalyzed the deep learning revolution and remains the standard source for pre-trained visual features.
2.2 Architecture Evolution
2.3 Choosing a Pre-trained Model
| Model | Parameters | Top-1 (ImageNet) | Best For |
|---|---|---|---|
| ResNet-18 | 11.7M | 69.8% | Quick prototyping, edge deployment |
| ResNet-50 | 25.6M | 76.1% | Good balance of speed/accuracy |
| EfficientNet-B0 | 5.3M | 77.1% | Mobile/embedded |
| EfficientNet-B4 | 19.3M | 82.9% | General-purpose |
| EfficientNet-B7 | 66.3M | 84.3% | Maximum accuracy |
| ConvNeXt-B | 89M | 83.8% | Transformer-like performance |
| ViT-B/16 | 86M | 77.9% | Vision Transformer baseline |
3. Feature Extraction vs. Fine-tuning
3.1 Feature Extraction
In feature extraction mode, the pre-trained backbone is treated as a fixed feature extractor. Only the newly added classification head is trained.
3.2 When to Use Each Approach
| Criterion | Feature Extraction | Fine-tuning |
|---|---|---|
| Training data | Less than 1,000 images | More than 1,000 images |
| Domain similarity | High (e.g., natural images) | Low (e.g., medical, satellite) |
| Compute budget | Low | High |
| Accuracy need | Moderate | High |
| Training time | Minutes to hours | Hours to days |
| Risk of overfitting | Low | Moderate to high |
4. Fine-tuning Strategies
4.1 Full Fine-tuning
Unfreeze all layers and train the entire network. Use this when you have sufficient data and compute.
4.2 Partial Fine-tuning
Freeze early layers, fine-tune later layers. This is the most common practical approach.
4.3 Gradual Unfreezing
Start by training only the classifier, then progressively unfreeze deeper layers. Popularized by the ULMFiT paper for NLP.
4.4 Layer Freezing Visualization
The diagram below shows how gradients flow through frozen and unfrozen layers:
5. Learning Rate Differentiation
5.1 Differential Learning Rates
Lower layers encode universal features that already generalize well — they need only slight adjustment. Higher layers encode more task-specific features that need more substantial adaptation. Using a single learning rate for all layers is suboptimal.
where is the learning rate for layer , is the total number of layers, and is a decay factor (typically 0.1 to 0.5).
5.2 Practical Implementation
5.3 Learning Rate Scheduling
Combined with differential LRs, schedulers adapt rates during training:
6. Domain Adaptation
When source and target domains differ (e.g., synthetic vs. real images, daytime vs. nighttime), domain adaptation aligns feature distributions.
6.1 Transfer Learning Concept Diagram
6.2 Maximum Mean Discrepancy (MMD)
Minimize the distance between source and target feature distributions:
where is a kernel-induced feature mapping, and are the number of source and target samples.
Intuition: If MMD is zero, the distributions are identical in the feature space. By minimizing MMD during training, we force the feature extractor to learn domain-invariant representations.
6.3 Adversarial Domain Adaptation
Train a domain discriminator to distinguish source vs. target features, while the feature extractor learns to fool it:
where is the feature generator, is the domain discriminator, and controls the trade-off.
7. Data Augmentation
Data augmentation is critical for transfer learning — it artificially expands small datasets and improves generalization.
7.1 Data Augmentation Examples
7.2 Implementation with torchvision
7.3 Advanced Augmentation: Mixup and CutMix
These techniques create virtual training examples by blending existing ones:
Mixup:
CutMix: Cut a patch from image and paste it onto image . Labels are mixed proportionally to the area ratio.
8. Implementation in PyTorch
8.1 Complete Fine-tuning Pipeline
8.2 Gradual Unfreezing with ULMFiT-style Training
8.3 Model Export and Inference
9. Fine-tuning Strategies Comparison
10. Common Pitfalls and Solutions
10.1 Catastrophic Forgetting
When fine-tuning erases source knowledge. Solutions: lower learning rates, freeze early layers, Elastic Weight Consolidation (EWC):
where is the Fisher information (importance) of parameter , and are optimal source parameters.
10.2 Negative Transfer
When source knowledge hurts target performance. Solutions: validate transfer benefits empirically; use similarity metrics to select source tasks.
10.3 Overfitting on Small Datasets
Solutions: strong augmentation, dropout, weight decay, early stopping, label smoothing.
10.4 Batch Normalization Issues
Frozen BN layers use source statistics, which may mismatch target data. Solution: use LayerNorm or train BN in eval mode with running statistics.
11. Key Takeaways
- Feature extraction (frozen backbone) is the fastest and safest approach for small datasets (less than 1K images per class)
- Fine-tuning with differential learning rates gives best results when data is sufficient; use smaller LR for pre-trained layers, larger LR for new layers
- Progressive unfreezing prevents catastrophic forgetting and stabilizes training by gradually unfreezing layers from top to bottom
- Domain adaptation (MMD, adversarial training) helps when source and target domains differ
- EWC penalizes changes to important parameters using Fisher information, preserving source knowledge
- Data augmentation (Mixup, CutMix, RandAugment) is essential for small datasets
- Negative transfer occurs when source and target tasks are too dissimilar; always validate empirically