Meta-Learning

Meta-learning, or "learning to learn," trains models to quickly adapt to new tasks with minimal data. Rather than learning a single task, the model acquires inductive biases that transfer across a distribution of tasks. This module covers optimization-based, metric-based, and model-based approaches with mathematical foundations.

1. Problem Formulation

Task Distribution

A task is drawn from a task distribution . Each task consists of:

• Support set: with classes and examples per class
• Query set:

The meta-objective minimizes expected loss over the task distribution:

where represents the task-specific adapted parameters.

Few-Shot Classification

The -way -shot setting: classes, support examples per class, and query examples.

where is the class prototype and is a distance metric.

2. Optimization-Based Meta-Learning

2.1 MAML: Model-Agnostic Meta-Learning

MAML (Finn et al., 2017) learns an initialization that can be quickly fine-tuned:

Inner loop (task-specific adaptation):

For gradient steps:

Outer loop (meta-update):

Second-order gradients: The meta-gradient involves:

This is computationally expensive: per task for the Hessian-vector product. In practice, we compute:

First-order MAML (FOMAML): Approximates by ignoring the Hessian:

This works surprisingly well in practice.

2.2 Reptile

Reptile (Nichol et al., 2018) is a first-order method:

where are the parameters after gradient steps on task .

Theorem: Reptile converges to a stationary point of where is the Hessian and is the gradient covariance across tasks.

Reptile avoids computing Hessian-vector products entirely, making it more memory-efficient than MAML.

2.3 Meta-SGD

Meta-SGD (Li et al., 2017) learns both the initialization and the learning rate:

where are per-parameter learning rates learned alongside .

2.4 ANIL: Almost No Inner Loop

ANIL (Raghu et al., 2019) splits the network into:

• Body (feature extractor): Meta-learned, fixed during adaptation
• Head (classifier): Adapted per task

The inner loop only updates the head, providing faster adaptation while maintaining MAML-level performance.

2.5 iMAML: Implicit MAML

iMAML (Finn et al., 2018) uses implicit differentiation:

The meta-gradient is computed implicitly via:

This avoids unrolling the optimization and reduces memory to .

3. Metric-Based Meta-Learning

3.1 Prototypical Networks

Prototypical Networks (Snell et al., 2017) represent each class by its mean embedding:

Class prototype:

Classification:

Euclidean distance yields prototypes as centroids:

Theorem: With Euclidean distance, Prototypical Networks minimize the distortion of the embedding space, clustering same-class examples around class centers.

3.2 Matching Networks

Matching Networks (Vinyals et al., 2016) use attention over the support set:

where the attention kernel is:

Full Context Embedding (FCE): Uses LSTM to encode the entire support set:

3.3 Relation Network

Relation Network (Sung et al., 2018) learns the distance metric:

where is a learned relation module (typically a small CNN).

3.4 Siamese Networks

Siamese Networks (Koch et al., 2015) use a shared encoder with a verification head:

4. Model-Based Meta-Learning

4.1 Memory-Augmented Neural Networks (MANN)

MANN (Santoro et al., 2016) uses external memory with NTM-style read/write:

Read:

where is the addressing weight.

Write:

4.2 Neural Processes

Neural Processes (Garnelo et al., 2018) model the conditional distribution:

where is a summary statistic.

ELBO:

4.3 Attentive Neural Processes

Attentive Neural Processes (Kim et al., 2019) use cross-attention:

where are the context points.

5. Task Augmentation and Construction

5.1 Task Construction

For image classification: sample a subset of classes, then sample K examples per class from a different subset of data.

5.2 Task Augmentation

Domain-specific augmentation: Apply transformations that preserve class identity within a task.

Implicit task augmentation: Add noise to task gradients during meta-update.

5.3 Task Distribution Learning

where parameterizes the task generator (e.g., class subsets, data augmentations).

6. Convergence Analysis

MAML Convergence

Theorem (Fallah et al., 2020): MAML converges to an -approximate first-order stationary point in iterations under standard assumptions.

Gradient complexity: where is the number of inner loop steps and is the Hessian computation cost.

Reptile Convergence

Theorem (Nichol et al., 2018): After iterations with step size :

7. Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F

class MAML(nn.Module):
    def __init__(self, model, inner_lr=0.01, inner_steps=5):
        super().__init__()
        self.model = model
        self.inner_lr = inner_lr
        self.inner_steps = inner_steps

    def inner_loop(self, support_x, support_y):
        fast_weights = list(self.model.parameters())
        for _ in range(self.inner_steps):
            logits = self.model.functional_forward(support_x, fast_weights)
            loss = F.cross_entropy(logits, support_y)
            grads = torch.autograd.grad(loss, fast_weights, create_graph=True)
            fast_weights = [w - self.inner_lr * g for w, g in zip(fast_weights, grads)]
        return fast_weights

    def forward(self, support_x, support_y, query_x):
        adapted_weights = self.inner_loop(support_x, support_y)
        query_logits = self.model.functional_forward(query_x, adapted_weights)
        return query_logits

    def meta_loss(self, batch):
        total_loss = 0
        for task in batch:
            support_x, support_y, query_x, query_y = task
            query_logits = self.forward(support_x, support_y, query_x)
            total_loss += F.cross_entropy(query_logits, query_y)
        return total_loss / len(batch)

class PrototypicalNetwork(nn.Module):
    def __init__(self, encoder, distance='euclidean'):
        super().__init__()
        self.encoder = encoder
        self.distance = distance

    def compute_prototypes(self, support_x, support_y, n_way):
        embeddings = self.encoder(support_x)
        prototypes = []
        for c in range(n_way):
            mask = support_y == c
            class_embeddings = embeddings[mask]
            prototypes.append(class_embeddings.mean(dim=0))
        return torch.stack(prototypes)

    def forward(self, support_x, support_y, query_x, n_way):
        prototypes = self.compute_prototypes(support_x, support_y, n_way)
        query_embeddings = self.encoder(query_x)
        
        if self.distance == 'euclidean':
            dists = torch.cdist(query_embeddings, prototypes)
            logits = -dists
        elif self.distance == 'cosine':
            sims = F.cosine_similarity(
                query_embeddings.unsqueeze(1), 
                prototypes.unsqueeze(0), dim=2
            )
            logits = sims
        
        return logits

class NeuralProcess(nn.Module):
    def __init__(self, x_dim, y_dim, r_dim, z_dim):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(x_dim + y_dim, 128),
            nn.ReLU(),
            nn.Linear(128, r_dim)
        )
        self.r_to_mu = nn.Linear(r_dim, z_dim)
        self.r_to_sigma = nn.Linear(r_dim, z_dim)
        self.decoder = nn.Sequential(
            nn.Linear(x_dim + z_dim, 128),
            nn.ReLU(),
            nn.Linear(128, y_dim * 2)
        )

    def encode(self, context_x, context_y):
        context = torch.cat([context_x, context_y], dim=-1)
        r = self.encoder(context).mean(dim=0)
        mu = self.r_to_mu(r)
        sigma = F.softplus(self.r_to_sigma(r))
        return mu, sigma

    def forward(self, context_x, context_y, target_x):
        mu, sigma = self.encode(context_x, context_y)
        z = mu + sigma * torch.randn_like(sigma)
        z_expanded = z.unsqueeze(0).expand(target_x.shape[0], -1)
        decoder_input = torch.cat([target_x, z_expanded], dim=-1)
        out = self.decoder(decoder_input)
        return out[..., :1], F.softplus(out[..., 1:])

8. SVG: MAML Inner/Outer Loop

9. SVG: Prototypical Network Classification

10. Advanced Topics

10.1 Meta-Learning with Heterogeneous Tasks

Different tasks may have different input/output dimensions. Solutions include:

• Task encoding: Map task metadata to hyperparameters
• Universal transformers: Dynamically adjust architecture per task
• Modular meta-learning: Compose task-specific modules

10.2 Gradient-Based Meta-Learning with Implicit Gradients

iMAML and Meta-SGD reduce the memory footprint from (unrolling) to (implicit differentiation).

10.3 Online Meta-Learning

Continuously learn from a stream of tasks without forgetting:

10.4 Multi-Task Meta-Learning

Joint optimization across tasks with meta-learned task-specific adaptations:

11. Comparison of Methods

12. Open Problems

• Scalability: MAML with second-order gradients scales poorly to large models
• Task distribution shift: Meta-test tasks may differ from meta-training
• Multi-modal meta-learning: Adapting across modalities (vision, language, robotics)
• Theoretical understanding: Why does meta-learning generalization work?
• Continual meta-learning: Avoiding catastrophic forgetting while learning new tasks

References

1. Finn, C., Abbeel, P., & Levine, S. (2017). Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. ICML.
2. Nichol, A., Achiam, J., & Schulman, J. (2018). On First-Order Meta-Learning Algorithms. arXiv:1803.02999.
3. Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical Networks for Few-shot Learning. NeurIPS.
4. Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., & Wierstra, D. (2016). Matching Networks for One Shot Learning. NeurIPS.
5. Garnelo, M., Rosenbaum, D., Maddison, C.J., et al. (2018). Conditional Neural Processes. ICML.
6. Finn, C., Yu, T., Zhang, T., Abbeel, P., & Levine, S. (2018). One-Shot Visual Imitation Learning via Meta-Learning. CoRL.
7. Raghu, A., Raghu, M., Bengio, S., & Vinyals, O. (2019). Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. ICLR.