Few-Shot and Zero-Shot Visual Learning

Introduction to Few-Shot Learning

Few-shot learning addresses the challenge of learning new visual concepts from very few labeled examples. While deep learning typically requires thousands of labeled images per class, few-shot methods aim to recognize new categories from as few as 1-5 examples. This capability is essential for applications where data collection is expensive (medical imaging), rare (defect detection), or privacy-restricted (personalized services).

The key insight is to learn a learning algorithm rather than learning specific categories. By training on a diverse set of tasks with few examples each, the model learns to quickly adapt to new tasks. This meta-learning approach treats each few-shot classification problem as a separate task, training the model to generalize across tasks rather than across examples within a single task.

Few-shot learning has enabled practical applications in medical imaging where rare diseases have very few documented cases, in retail where new product categories emerge frequently, and in content moderation where new types of harmful content appear regularly. The ability to rapidly adapt to new concepts with minimal labeled data makes few-shot learning particularly valuable in dynamic environments where the class distribution evolves over time.

Prototypical Networks

Prototypical Networks learn a metric space where classification is performed by computing distances to class prototypes. Each class prototype is the mean of the embedded support examples for that class. Classification of a query image is performed by computing distances to all class prototypes and applying softmax.

For a K-way N-shot problem, the prototype for class is computed as:

Where each parameter means:

• is the prototype vector for class
• is the set of support examples belonging to class
• is the embedding of support example by the encoder network
• is the number of shots (examples per class)
• The mean operation creates a representative vector for each class

The classification probability for query is computed using softmax over negative distances:

Where each parameter means:

• is the embedding of the query image
• is the distance metric (typically squared Euclidean)
• is the prototype for class
• The negative distance ensures closer prototypes have higher probability
• Softmax normalizes probabilities to sum to 1

The Euclidean distance metric is often preferred over cosine similarity for prototypical networks because it better captures the geometric structure of the embedding space learned during meta-training. Prototypical networks achieve strong performance with simple architecture and efficient training, making them a popular choice for practical few-shot learning applications across diverse domains including medical imaging, satellite imagery analysis, and industrial defect detection.

Matching Networks

Matching Networks use an attention mechanism over support examples weighted by their similarity to the query. Unlike prototypical networks that average support examples, matching Networks compute a weighted combination where the weights depend on the query-support similarity. This allows the network to learn task-specific adaptations rather than using a fixed encoder, enabling more flexible few-shot classification.

The attention mechanism for classification is:

Where each parameter means:

• is the query embedding (full context embedding)
• is the support example embedding
• is the cosine similarity
• The attention weights sum to 1 over all support examples
• The classifier output is

Matching Networks also introduce a Full Context Embedding (FCE) that uses LSTM to encode each example with awareness of the entire support set. This allows the network to learn task-specific adaptations rather than using a fixed encoder, enabling more flexible few-shot classification that can handle varying task distributions and class relationships.

Python Implementation: Prototypical Network

import torch
import torch.nn as nn
import torch.nn.functional as F


class PrototypicalNetwork(nn.Module):
    def __init__(self, encoder_dim=64):
        super(PrototypicalNetwork, self).__init__()
        self.encoder = nn.Sequential(
            self._make_block(3, 64),
            self._make_block(64, 64),
            self._make_block(64, 64),
            self._make_block(64, encoder_dim),
        )

    def _make_block(self, in_ch, out_ch):
        return nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2)
        )

    def forward(self, support_images, support_labels, query_images):
        n_way = len(torch.unique(support_labels))
        support_embeddings = self.encoder(support_images)
        prototypes = []
        for c in range(n_way):
            mask = support_labels == c
            class_embeddings = support_embeddings[mask]
            prototype = class_embeddings.mean(dim=0)
            prototypes.append(prototype)
        prototypes = torch.stack(prototypes)
        query_embeddings = self.encoder(query_images)
        dists = torch.cdist(query_embeddings, prototypes)
        log_probs = F.log_softmax(-dists, dim=1)
        return log_probs


def train_prototypical(model, meta_loader, optimizer, num_epochs=100):
    model.train()
    for epoch in range(num_epochs):
        total_loss = 0
        total_acc = 0
        for episode in meta_loader:
            support_imgs, support_labels = episode['support']
            query_imgs, query_labels = episode['query']
            log_probs = model(support_imgs, support_labels, query_imgs)
            loss = F.nll_loss(log_probs, query_labels)
            preds = log_probs.argmax(dim=1)
            acc = (preds == query_labels).float().mean()
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
            total_acc += acc.item()
        avg_loss = total_loss / len(meta_loader)
        avg_acc = total_acc / len(meta_loader)
        print(f"Epoch {epoch+1}: Loss={avg_loss:.4f}, Acc={avg_acc:.4f}")

Common Challenges

1. Task Distribution Shift: The meta-training task distribution may differ from the deployment task distribution, leading to poor generalization to truly novel categories.

2. Domain Gap: Few-shot learning assumes train and test tasks come from the same domain. Cross-domain few-shot learning requires additional techniques like domain adaptation, meta-domain generalization, or prompt-based adaptation.

3. Imbalanced Classes: Real-world few-shot scenarios often have imbalanced class distributions, requiring specialized loss functions, balanced sampling strategies, or reweighting mechanisms to handle long-tail distributions.

4. Computational Cost: Meta-training requires sampling many tasks and computing gradients through multiple steps, making training computationally expensive. Efficient meta-learning algorithms like Meta-SGD and ANML reduce the inner-loop computation.

5. Feature Quality: The quality of learned representations directly impacts few-shot performance. Self-supervised pre-training can significantly improve few-shot results by providing strong initial features without requiring labels.

6. Task Distribution: The meta-training task distribution may not cover all possible test scenarios. Task augmentation, task scheduling, and diverse data sampling help improve coverage of the task space.

7. Scalability: Scaling to thousands of classes and millions of images while maintaining few-shot performance requires efficient memory banks, prototype caching, and hierarchical classification strategies that balance representational capacity with computational efficiency.

8. Interpretability: Understanding why meta-learning models make specific predictions is challenging. Prototype visualization and attention analysis provide insights into the decision-making process, helping build trust for clinical and high-stakes applications.

Few-shot learning uses specialized evaluation protocols that simulate the few-shot scenario during testing. The N-way K-shot protocol evaluates classification performance with N classes and K examples per class in the support set. A typical evaluation uses 5-way 1-shot and 5-way 5-shot settings with 600 or 1000 test episodes for statistical significance. Episode-based evaluation ensures that the model's performance reflects its ability to learn from limited data rather than memorizing specific categories.

Case Study: Medical Rare Disease Classification

A hospital deployed prototypical networks for classifying rare skin conditions from dermoscopy images. The system was meta-trained on 100 common skin conditions with 5-10 examples each, then deployed for 15 rare conditions with only 3 examples per class. The model achieved 78% accuracy on the rare conditions, compared to 45% for standard transfer learning. Radiologists using the system as a diagnostic aid reported a 32% reduction in misdiagnosis rates for rare conditions. The system processes images in 0.3 seconds, enabling real-time clinical decision support. After 6 months of deployment, the hospital documented 23 cases where the system correctly identified conditions that were initially misclassified by human experts. The few-shot approach has been扩展 to other medical specialties including pathology and ophthalmology, where labeled data for rare diseases is particularly scarce. The hospital reports that the system has reduced the average time to diagnosis for rare conditions from 6 weeks to 2 weeks, significantly improving patient outcomes.

Key Takeaways

• Few-shot learning trains models to learn new concepts from 1-5 labeled examples
• Prototypical Networks compute class prototypes as means of support embeddings
• Matching Networks use attention mechanisms weighted by query-support similarity
• Episodic training simulates few-shot scenarios during meta-training
• The choice of distance metric significantly impacts few-shot classification accuracy
• Self-supervised pre-training provides strong feature representations for few-shot tasks
• Meta-learning algorithms optimize the learning process across diverse task distributions
• Prototypical networks are simple yet effective for many few-shot classification scenarios
• The choice of distance metric significantly impacts few-shot classification accuracy and must be carefully selected