Advanced Topics

Self-Supervised Learning — Learning Without Labels

Master self-supervised learning techniques that leverage unlabeled data to learn powerful representations. The foundation of modern NLP and computer vision.

Contrastive Learning — Learning by comparing similar and dissimilar examples
Masked Language Modeling — BERT-style pre-training on text
SimCLR — Simple framework for contrastive learning of visual representations

"The best way to learn is to teach yourself."

📋 Prerequisites

● Deep Learning: Neural networks, backpropagation, CNNs, Transformers
● Computer Vision: Image classification, convolutional architectures (ResNet, ViT)
● NLP Basics: Tokenization, word embeddings, language modeling concepts
● Python & PyTorch: Data augmentation (torchvision), model training
● Linear Algebra: Matrix operations, cosine similarity, distance metrics

🎯 Learning Objectives

✓Understand why self-supervised learning is the foundation of modern AI

✓Implement SimCLR contrastive learning pipeline

✓Apply masked language modeling (BERT/MAE) for pre-training

✓Compare contrastive vs non-contrastive vs generative methods

✓Design pre-training and fine-tuning pipelines

✓Evaluate pre-trained representations on downstream tasks

Self-Supervised Learning — Complete Guide

Self-supervised learning creates labels from the data itself, enabling training on massive unlabeled datasets.

Self-Supervised Learning Landscape

Key Formulas Reference

Key Formulas — Self-Supervised Learning

NT-Xent Loss (SimCLR):

Where τ = temperature parameter, k ∈ {1,...,2N} for a batch of N positive pairs, and sim(·,·) is cosine similarity.

Masked Language Modeling (BERT):

Predict masked tokens x_m given the context x_¬M. Typically 15% of tokens are masked.

BYOL Loss:

Student network q_θ predicts teacher output without negative pairs. Teacher uses stop-gradient.

CLIP Contrastive Loss:

Symmetric cross-entropy over image-text similarity matrix. i2t = image-to-text, t2i = text-to-image.

MAE Reconstruction Loss:

MSE loss computed only over masked patches. |M| = number of masked patches, typically 75%.

Why Self-Supervised?

Contrastive Learning (SimCLR)

Masked Language Modeling (BERT/MAE)

BYOL: Bootstrap Your Own Latent

Fine-Tuning Strategies

Python Implementation Example

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.transforms as T
from torchvision.models import resnet50

class SimCLR(nn.Module):
    def __init__(self, backbone, projection_dim=128):
        super().__init__()
        self.encoder = backbone
        encoder_dim = backbone.fc.in_features
        backbone.fc = nn.Identity()
        
        self.projector = nn.Sequential(
            nn.Linear(encoder_dim, encoder_dim),
            nn.ReLU(),
            nn.Linear(encoder_dim, projection_dim)
        )
    
    def forward(self, x):
        h = self.encoder(x)
        z = self.projector(h)
        return h, z

class NTXentLoss(nn.Module):
    def __init__(self, temperature=0.5):
        super().__init__()
        self.temperature = temperature
    
    def forward(self, z_i, z_j):
        batch_size = z_i.shape[0]
        z = torch.cat([z_i, z_j], dim=0)
        z = F.normalize(z, dim=1)
        
        sim = torch.mm(z, z.t()) / self.temperature
        mask = torch.eye(2 * batch_size, dtype=torch.bool)
        sim.masked_fill_(mask, -float('inf'))
        
        labels = torch.cat([
            torch.arange(batch_size, 2 * batch_size),
            torch.arange(batch_size)
        ])
        
        return F.cross_entropy(sim, labels)

# Augmentation pipeline
transform = T.Compose([
    T.RandomResizedCrop(224),
    T.RandomHorizontalFlip(),
    T.ColorJitter(0.8, 0.8, 0.8, 0.2),
    T.RandomGrayscale(p=0.2),
    T.GaussianBlur(kernel_size=23),
    T.ToTensor(),
])

# Training loop
model = SimCLR(resnet50()).cuda()
criterion = NTXentLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=3e-4)

for epoch in range(100):
    for images in dataloader:
        # Two augmented views of same image
        view1 = transform(images)
        view2 = transform(images)
        
        h1, z1 = model(view1.cuda())
        h2, z2 = model(view2.cuda())
        
        loss = criterion(z1, z2)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Real-World Applications

🌍 Real-World Applications of Self-Supervised Learning

1. Large Language Models (GPT, LLaMA)

GPT-4, LLaMA, and other LLMs use next-token prediction (a self-supervised task) to pre-train on trillions of tokens from the internet. This enables zero-shot and few-shot capabilities across thousands of NLP tasks without task-specific training.

2. Vision Transformers (ViT, DINO)

DINO and MAE pre-train ViT models on millions of unlabeled images. These models achieve state-of-the-art on image classification, object detection, and segmentation with minimal labeled data. DINO's self-attention maps learn object segmentation without any labels.

3. Multimodal Models (CLIP)

CLIP pre-trains on 400M image-text pairs using contrastive learning. It learns to match images with their textual descriptions, enabling zero-shot image classification, image search, and visual question answering without task-specific fine-tuning.

4. Medical Imaging

Self-supervised pre-training on unlabeled medical images (X-rays, MRIs, CT scans) learns rich visual representations. Fine-tuning with just 100 labeled images achieves performance comparable to supervised models trained on thousands of images, crucial for rare diseases.

5. Speech Recognition (Wav2Vec)

Wav2Vec 2.0 pre-trains on raw audio waveforms using masked prediction. It learns speech representations that transfer to low-resource languages with just 10 minutes of labeled data, enabling speech recognition in 100+ languages.

6. Protein Structure Prediction (AlphaFold)

Self-supervised learning on protein sequences enables AlphaFold to predict 3D protein structures. Pre-training on millions of sequences learns evolutionary patterns and structural constraints, solving a 50-year grand challenge in biology.

Common Mistakes & How to Avoid Them

⚠️ Common Mistakes & How to Avoid Them

1
Weak Data Augmentations:
Weak augmentations (only horizontal flip) don't create enough variation for contrastive learning. Use strong augmentations: random crop + color jitter + Gaussian blur + grayscale. The augmentation pipeline defines what "similar" means.
2
Too Small Batch Size:
Contrastive learning needs large batches (256-8192) for enough negative pairs. Small batches collapse representations. Use gradient accumulation if GPU memory is limited. MoCo uses a queue as an alternative to large batches.
3
Wrong Temperature (τ):
Temperature controls the sharpness of the softmax. Too low (τ=0.01) causes numerical instability; too high (τ=1.0) makes learning too uniform. Start with τ=0.1-0.5 for NT-Xent loss.
4
Removing the Projection Head:
The projection head is crucial during training (removes information not useful for the pretext task) but should be discarded for downstream use. Use the representation before the projection head (z, not h) for fine-tuning.
5
Ignoring Downstream Task Alignment:
Pre-training on a different domain than the downstream task reduces benefits. Align pre-training data with downstream task. Use domain-specific augmentations and architectures.
6
Representation Collapse:
All outputs become the same constant (collapsed representation). Monitor embedding diversity during training. Use stop-gradient (BYOL), asymmetric networks, or variance regularization to prevent collapse.

Interview Questions

💬 Interview Questions — Self-Supervised Learning

Q1: What is self-supervised learning?

Self-supervised learning creates labels from the data itself (pseudo-labels) to train models on unlabeled data. Unlike supervised learning (human labels) or unsupervised learning (no labels at all), SSL uses the structure of data as supervision — e.g., predicting masked words, next token, or contrasting image views.

Q2: Why is batch size important in contrastive learning?

In SimCLR, each positive pair has 2(N-1) negative pairs from the batch. Larger batches provide more negatives, making the contrastive task harder and representations better. Optimal performance typically requires batch sizes of 4096-8192. MoCo uses a queue of 65536 negatives as an alternative.

Q3: What is the difference between SimCLR and BYOL?

SimCLR uses negative pairs and the NT-Xent loss to push dissimilar representations apart. BYOL uses no negative pairs — instead, a student network learns to predict a teacher network's output (EMA of student). BYOL avoids collapse through architectural asymmetry and stop-gradient, often achieving better performance.

Q4: Why does BERT mask 15% of tokens?

Masking 15% balances training signal (too few = slow learning) with context availability (too much = insufficient context to predict). The 80-10-10 strategy (80% [MASK], 10% random, 10% original) prevents the model from expecting only [MASK] tokens at fine-tuning time, since real text never contains [MASK].

Q5: What is the role of the projection head?

The projection head (MLP after the encoder) transforms representations to the space where the contrastive loss is applied. It removes information not useful for the pretext task. At test time, discard the projection head and use the encoder's output — it contains richer, more transferable features.

Q6: How does CLIP learn vision-language alignment?

CLIP trains an image encoder and text encoder to match images with their descriptions using contrastive learning. Given a batch of N image-text pairs, it learns to match each image with its correct text (positive) while treating others as negatives. This enables zero-shot transfer — classify any image using natural language descriptions.

Q7: When should you use self-supervised vs supervised pre-training?

Use self-supervised when you have abundant unlabeled data but limited labels (medical imaging, low-resource NLP). Use supervised pre-training when you have large labeled datasets (ImageNet) and the downstream task is similar. Self-supervised is better for domain shift, supervised is better when tasks closely match.

Practice Exercise

🏋️ Practice Exercise — SimCLR Implementation

Challenge:

Implement a complete SimCLR pipeline and evaluate on CIFAR-10:

Implement SimCLR with ResNet-18 backbone
Build augmentation pipeline: random crop, color jitter, Gaussian blur, grayscale
Train with NT-Xent loss for 200 epochs (batch size 512)
Evaluate via linear probe: freeze encoder, train linear classifier on CIFAR-10 train set
Compare: (a) Random init baseline, (b) SimCLR pre-train, (c) Supervised pre-train

Expected Results:

Random init + linear probe: ~55% accuracy
SimCLR pre-train + linear probe: ~78% accuracy
Supervised pre-train + linear probe: ~85% accuracy
SimCLR + fine-tune: ~88% accuracy

Comparison Table

📊 Self-Supervised Learning Methods Comparison

Method	SimCLR	BYOL	BERT	CLIP
Type	Contrastive	Non-contrastive	Masked prediction	Contrastive (cross-modal)
Modality	Vision	Vision	NLP	Vision + Language
Negative Pairs	Required	Not needed	N/A	Required
Batch Size Needed	Large (4096+)	Medium (256-1024)	N/A	Large (32768)
Pre-training Data	ImageNet (1M)	ImageNet-1K/22K	BooksCorpus, Wikipedia	400M image-text pairs
Collapse Prevention	Negative pairs	EMA + asymmetric arch	N/A (generative)	Negative pairs
Best For	Visual representations	Efficient pre-training	NLP tasks	Zero-shot transfer

Key Takeaways

📌 Key Takeaways — Self-Supervised Learning

▸ Self-supervised learning creates labels from data — no human annotation needed
▸ Contrastive learning learns by comparing pairs (SimCLR, MoCo, CLIP)
▸ Non-contrastive methods avoid negative pairs (BYOL, DINO, SimSiam)
▸ Masked modeling learns by predicting hidden parts (BERT, GPT, MAE)
▸ Pre-train + fine-tune is the dominant paradigm in modern ML
▸ Data augmentation defines the learning signal in contrastive methods
▸ Projection heads are crucial for training, representations come before them
▸ Self-supervised learning enables foundation models (GPT, LLaMA, ViT)
▸ CLIP learns vision-language alignment via contrastive pre-training
▸ Large batches (4096+) and strong augmentations are critical for contrastive methods
▸ Representation collapse is prevented by negative pairs, EMA, or architectural asymmetry
▸ Domain-specific pre-training (medical, speech, proteins) yields best transfer performance
▸ Linear probing is a fast way to evaluate pre-trained representations without full fine-tuning

What to Learn Next

-> BERT and Encoder Models — Complete Guide Learn about bert and encoder models — complete guide.

-> GPT Architecture — Decoder-Only Transformers Complete Guide Learn about gpt architecture — decoder-only transformers complete guide.

-> Transfer Learning — Pre-trained Models Complete Guide Learn about transfer learning — pre-trained models complete guide.

-> Transformers — Attention Is All You Need Complete Guide Learn about transformers — attention is all you need complete guide.

-> Meta-Learning — Learning to Learn Learn about meta-learning — learning to learn.

-> GANs — Generative Adversarial Networks Complete Guide Learn about gans — generative adversarial networks complete guide.

Self-Supervised Learning — Pre-training Revolution