Self-Supervised Learning: Pretext Tasks and Beyond

Module: Machine Learning | Difficulty: Advanced

Pretext Tasks

BERT: Masked Language Modeling

GPT: Autoregressive

Theory

where is the learned representation.

import torch
import torch.nn as nn

class PretextTask(nn.Module):
    def __init__(self, encoder, n_rotations=4):
        super().__init__()
        self.encoder = encoder
        self.rotation_head = nn.Linear(encoder.output_dim, n_rotations)
    def forward(self, x, rotation_label=None):
        features = self.encoder(x)
        rotation_pred = self.rotation_head(features)
        if rotation_label is not None:
            loss = nn.functional.cross_entropy(rotation_pred, rotation_label)
            return loss, features
        return rotation_pred, features

Research Insight: Self-supervised pre-training can match supervised pre-training on ImageNet with 10x less labeled data. The key insight is that pretext tasks that require understanding high-level semantics transfer better to downstream tasks.