Self-Supervised Learning

Self-supervised learning (SSL) eliminates the dependency on human-annotated labels by constructing auxiliary pretext tasks from unlabeled data. The learned representations transfer downstream to classification, detection, generation, and reinforcement learning. This module covers the three dominant paradigms—contrastive, masked, and joint embedding—along with their mathematical foundations.

1. Overview and Taxonomy

Self-Supervised Learning
├── Contrastive Methods
│   ├── Instance Discrimination (InstDisc, CPC)
│   ├── Memory Bank (MoCo v1/v2)
│   ├── No Negatives (BYOL, SimSiam, VICReg)
│   └── Clustering (SwAV, DINO)
├── Masked Methods
│   ├── Masked Image Modeling (MAE, BEiT)
│   ├── Masked Language Modeling (BERT-style)
│   └── Masked Patch Prediction
└── Joint Embedding Predictive (JEPA)
    ├── Video JEPA (V-JEPA)
    └── Image JEPA (I-JEPA)

2. The Instance Discrimination Hypothesis

The foundational premise: every image instance defines its own class. Given an unlabeled dataset , we construct positive pairs through augmentations and treat all other instances as negatives.

Augmentation Distribution

Define an augmentation pipeline composed of random crops, color jitter, Gaussian blur, solarization, and horizontal flips. For an image , a positive pair is:

The key insight is that must preserve semantic content while creating sufficient low-level variation to prevent collapse.

Non-Contrastive Objective

Recent methods show that explicit negative pairs are unnecessary. Instead, alignment and uniformity on a hypersphere suffice:

where maps inputs to the unit hypersphere.

3. Contrastive Learning Frameworks

3.1 SimCLR: Simple Framework for Contrastive Learning

SimCLR (Chen et al., 2020) establishes the modern contrastive learning recipe:

Architecture: ResNet encoder followed by a projection head (2-layer MLP).

NT-Xent Loss (Normalized Temperature-scaled Cross-Entropy):

For a batch of augmented views :

where is cosine similarity and is the temperature.

The total loss is .

Temperature Analysis: The temperature controls the penalty for hard negatives. As , the loss concentrates on the hardest negatives. As , all negatives contribute equally. Optimal is typically –.

3.2 MoCo: Momentum Contrast

MoCo (He et al., 2020) decouples batch size from the number of negatives via a momentum-updated queue:

Queue-based dictionary: Maintain a FIFO queue of size with encoded keys.

Momentum update:

where is the query encoder and is the key encoder.

InfoNCE Loss:

MoCo v2 adds the projection head and stronger augmentations from SimCLR, achieving comparable results with smaller batch sizes.

3.3 BYOL: Bootstrap Your Own Latent

BYOL (Grill et al., 2020) eliminates negative pairs entirely:

Architecture: Online network (encoder + predictor) and target network (EMA of online).

Predictor: A 2-layer MLP in the online network only.

Loss:

where and .

Why no collapse? The asymmetry introduced by the predictor prevents trivial solutions. The target network's slow update creates a moving target that the online network cannot "chase" to a degenerate solution.

3.4 DINO: Self-Distillation with No Labels

DINO (Caron et al., 2021) combines self-distillation with multi-crop augmentation:

Student receives global + local crops; Teacher receives only global crops.

Cross-entropy with centering:

Centering prevents mode collapse:

The center is subtracted from the teacher's output before softmax.

DINOv2 extends this with:

• Layer-wise distillation at multiple depths
• Register tokens for spatial correspondence
• Curated pseudo-labels from clustering

3.5 SwAV: Sinkhorn-Knopp Assignment

SwAV (Caron et al., 2020) assigns prototypes via online optimal transport:

Sinkhorn-Knopp computes the assignment such that:

The loss uses swapped prediction:

4. Redundancy Reduction Methods

4.1 Barlow Twins

Barlow Twins (Zbontar et al., 2021) uses the cross-correlation matrix between batch representations:

Cross-correlation:

Loss:

The invariance term pushes diagonal entries to 1. The redundancy reduction term pushes off-diagonal entries to 0, decorrelating representation dimensions.

4.2 VICReg: Variance-Invariance-Covariance Regularization

VICReg (Bardes et al., 2022) decomposes the loss into three terms:

Variance (prevents collapse by ensuring each dimension has nonzero variance):

Invariance (enforces representation invariance across views):

where denotes the mean-centered representations.

Covariance (decorrelates dimensions):

5. Masked Autoencoders

5.1 MAE: Masked Autoencoder

MAE (He et al., 2022) follows the encoder-decoder paradigm with asymmetric masking:

Encoding: Only visible patches (25% of total) are processed by the encoder:

Decoding: The full sequence where are learnable mask tokens is processed by a lightweight decoder:

Loss (reconstruction on masked patches only):

where is the set of masked patch indices.

Asymmetric design rationale: The encoder only processes visible patches, reducing compute by ~75%. The decoder is lightweight (2 transformer blocks vs. 12 encoder blocks).

5.2 BEiT: Bidirectional Encoder Representation from Image Transformers

BEiT (Bao et al., 2022) uses discrete tokens as reconstruction targets:

1. Tokenizer (dVAE): produces visual tokens
2. Pretext: Predict from
3. Loss: Cross-entropy over masked positions

5.3 EMAE: Exponential Moving Average Autoencoder

Combines MAE with exponential moving average teacher for improved stability.

6. Joint Embedding Predictive Architecture (JEPA)

JEPA (LeCun, 2022) predicts representations in latent space rather than pixel space:

Architecture: Encoder , predictor , target encoder .

Objective:

where is the input and is a related view (e.g., different temporal segment).

I-JEPA (Image JEPA): Predicts masked patch representations from visible patches using an exponential moving average target encoder.

V-JEPA (Video JEPA): Spatiotemporal joint embedding for video understanding, predicting future/missing spatiotemporal regions.

Theoretical advantage: By operating in latent space, JEPA avoids the "pixel prediction bottleneck"—the model need not waste capacity on imperceptible details.

7. Theoretical Foundations

7.1 Information-Theoretic View

The contrastive objective maximizes a lower bound on the mutual information :

This is the InfoNCE bound (Oord et al., 2018). With perfect discrimination (), the bound approaches .

7.2 Collapse Prevention

Dimensional collapse occurs when representations span a lower-dimensional subspace. Representation collapse occurs when all outputs converge to a constant.

Sufficient conditions for non-collapse:

• Negative pairs (contrastive): Prevent collapse by repelling all non-positive pairs
• Asymmetry (BYOL, DINO): Prevents trivial solutions by breaking gradient flow symmetry
• Redundancy reduction (Barlow Twins, VICReg): Explicitly decorrelate dimensions
• Stop-gradient (SimSiam): The target network's stop-gradient prevents chasing degenerate solutions

7.3 Uniformity and Alignment

Theorem (Wang & Isola, 2020): The contrastive loss decomposes as:

Optimal representations are both aligned (close positive pairs) and uniform (evenly distributed on the hypersphere).

8. Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F

class SimCLR(nn.Module):
    def __init__(self, encoder, projection_dim=128, hidden_dim=2048):
        super().__init__()
        self.encoder = encoder
        self.projector = nn.Sequential(
            nn.Linear(encoder.out_features, hidden_dim),
            nn.BatchNorm1d(hidden_dim),
            nn.ReLU(inplace=True),
            nn.Linear(hidden_dim, projection_dim),
        )

    def forward(self, x1, x2):
        h1 = self.encoder(x1)
        h2 = self.encoder(x2)
        z1 = F.normalize(self.projector(h1), dim=1)
        z2 = F.normalize(self.projector(h2), dim=1)
        return z1, z2

def nt_xent_loss(z1, z2, temperature=0.07):
    batch_size = z1.shape[0]
    z = torch.cat([z1, z2], dim=0)  # (2N, d)
    sim = torch.mm(z, z.T) / temperature  # (2N, 2N)
    mask = ~torch.eye(2 * batch_size, dtype=bool, device=z.device)
    labels = torch.cat([
        torch.arange(batch_size, 2 * batch_size),
        torch.arange(0, batch_size)
    ]).to(z.device)
    sim = sim.masked_select(mask).view(2 * batch_size, -1)
    loss = F.cross_entropy(sim, labels)
    return loss

class VICReg(nn.Module):
    def __init__(self, encoder, projection_dim=2048, 
                 variance_target=1.0, covariance_weight=0.04):
        super().__init__()
        self.encoder = encoder
        self.projector = nn.Sequential(
            nn.Linear(encoder.out_features, projection_dim),
            nn.BatchNorm1d(projection_dim),
            nn.ReLU(),
            nn.Linear(projection_dim, projection_dim),
        )
        self.variance_target = variance_target
        self.covariance_weight = covariance_weight

    def forward(self, x1, x2):
        z1 = self.projector(self.encoder(x1))
        z2 = self.projector(self.encoder(x2))
        
        # Invariance loss
        inv_loss = F.mse_loss(z1, z2)
        
        # Variance loss
        std_z1 = torch.sqrt(z1.var(dim=0) + 1e-4)
        std_z2 = torch.sqrt(z2.var(dim=0) + 1e-4)
        var_loss = torch.mean(F.relu(self.variance_target - std_z1)) + \
                   torch.mean(F.relu(self.variance_target - std_z2))
        
        # Covariance loss
        z1 = z1 - z1.mean(dim=0)
        z2 = z2 - z2.mean(dim=0)
        cov_z1 = (z1.T @ z1) / (z1.shape[0] - 1)
        cov_z2 = (z2.T @ z2) / (z2.shape[0] - 1)
        cov_loss = (off_diagonal(cov_z1).pow(2).sum() + 
                    off_diagonal(cov_z2).pow(2).sum()) / z1.shape[1]
        
        return inv_loss + var_loss + self.covariance_weight * cov_loss

def off_diagonal(x):
    n, m = x.shape
    assert n == m
    return x.flatten()[:-1].reshape(n - 1, n + 1)[:, 1:].flatten()

class MAE(nn.Module):
    def __init__(self, encoder, decoder_dim=512, decoder_depth=8, 
                 mask_ratio=0.75, patch_size=16):
        super().__init__()
        self.encoder = encoder
        self.patch_size = patch_size
        self.mask_ratio = mask_ratio
        self.decoder = TransformerDecoder(
            dim=decoder_dim, depth=decoder_depth
        )
        self.mask_token = nn.Parameter(torch.randn(1, 1, decoder_dim))
        self.prediction_head = nn.Linear(decoder_dim, patch_size ** 2 * 3)
        self.projection = nn.Linear(encoder.out_features, decoder_dim)

    def random_masking(self, x, mask_ratio):
        B, N, D = x.shape
        num_keep = int(N * (1 - mask_ratio))
        noise = torch.rand(B, N, device=x.device)
        ids_shuffle = noise.argsort(dim=1)
        ids_restore = ids_shuffle.argsort(dim=1)
        ids_keep = ids_shuffle[:, :num_keep]
        x_masked = torch.gather(x, dim=1, index=ids_keep.unsqueeze(-1).expand(-1, -1, D))
        return x_masked, ids_restore

    def forward(self, x):
        # Encode visible patches only
        x_masked, ids_restore = self.random_masking(x, self.mask_ratio)
        z = self.encoder(x_masked)
        z = self.projection(z)
        
        # Decode all positions
        B = z.shape[0]
        mask_tokens = self.mask_token.expand(B, ids_restore.shape[1] - z.shape[1], -1)
        z_full = torch.cat([z, mask_tokens], dim=1)
        z_full = torch.gather(z_full, dim=1, 
                              index=ids_restore.unsqueeze(-1).expand(-1, -1, z.shape[-1]))
        pred = self.prediction_head(z_full)
        
        return pred, ids_restore

9. Scaling Laws for Self-Supervised Learning

Research has identified several scaling relationships:

Data scaling: where – depending on the method and downstream task.

Model scaling: SSL benefits more from larger models than supervised learning at the same data scale, as larger models are more resistant to collapse.

Augmentation scaling: The diversity and strength of augmentations follow diminishing returns. After a critical augmentation strength, performance plateaus or degrades.

where is asymptotic performance, is data size, is model size, and captures augmentation effects.

10. Comparison Across Methods

11. SVG: Self-Supervised Pipeline

12. SVG: Contrastive Learning Pairs

13. Open Problems and Future Directions

• Data efficiency: How to reduce the massive pretraining data requirements?
• Task-specific fine-tuning: Can we avoid full fine-tuning and use lightweight adaptation?
• Multimodal SSL: CLIP, ALIGN, and their scaling to more modalities
• Temporal SSL: Learning from video and sequential data at scale
• World models: JEPA as a foundation for predictive world models (LeCun's "Path to Autonomous Intelligence")

References

1. Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. ICML.
2. He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020). Momentum Contrast for Unsupervised Visual Representation Learning. CVPR.
3. Grill, J.-B., Strub, F., Altché, F., et al. (2020). Bootstrap Your Own Latent. NeurIPS.
4. Caron, M., Touvron, H., Misra, I., et al. (2021). Emerging Properties in Self-Supervised Vision Transformers. ICCV.
5. Zbontar, J., Jing, L., Misra, I., LeCun, Y., & Deny, S. (2021). Barlow Twins: Self-Supervised Learning via Redundancy Reduction. ICML.
6. Bardes, A., Ponce, J., & LeCun, Y. (2022). VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. ICLR.
7. He, K., Chen, X., Xie, S., et al. (2022). Masked Autoencoders Are Scalable Vision Learners. CVPR.
8. Assran, M., Duval, Q., Misra, I., et al. (2023). Masked Image Modeling with Denoising Contrast. ICLR.