Domain Adaptation & Generalization

Domain adaptation addresses the distribution shift between training (source) and deployment (target) domains. This module covers theoretical foundations, importance weighting, adversarial methods, and out-of-distribution generalization.

1. Problem Formulation

Domain Definition

A domain consists of a feature space and a marginal distribution .

Source domain: Target domain:

Types of Shift

2. Theoretical Foundations

2.1 H-Divergence

Ben-David et al. (2010) bound target error via:

where:

• : Source error of classifier
• : -divergence between domains
• : Ideal joint error

2.2 Maximum Mean Discrepancy (MMD)

where is a kernel feature map.

Kernel MMD: Using :

2.3 Wasserstein Distance

3. Importance Weighting

3.1 Covariate Shift Correction

Under covariate shift, the target risk is:

Density ratio estimation:

3.2 Kernel Mean Matching (KMM)

Minimize the discrepancy between weighted source and target means:

subject to and .

3.3 Moment Matching

Match higher-order moments:

4. Adversarial Domain Adaptation

4.1 DANN: Domain-Adversarial Neural Network

Ganin et al. (2016) use adversarial training:

Architecture: Feature extractor , label predictor , domain classifier .

Objective:

where:

• : Label prediction loss (source only)
• : Domain classification loss (binary: source vs target)

Gradient Reversal Layer (GRL): During backpropagation, multiply gradient by :

This encourages the feature extractor to produce domain-invariant features.

4.2 DANN Training Algorithm

1. Sample source batch, compute
2. Sample target batch, compute
3. Update to minimize (domain classifier learns to distinguish)
4. Update to minimize (features become invariant)
5. Update to minimize

4.3 Gradient Reversal Layer

class GradientReversalFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, alpha):
        ctx.alpha = alpha
        return x.clone()
    
    @staticmethod
    def backward(ctx, grad_output):
        return -ctx.alpha * grad_output, None

5. Domain-Invariant Feature Learning

5.1 CORAL: Correlation Alignment

Align second-order statistics:

where are covariance matrices of source and target features.

5.2 MMD Distance

Deep CORAL adds MMD to the loss:

5.3 Batch Normalization Adaptation

Transductive Batch Norm: Use target statistics for batch normalization layers:

6. Out-of-Distribution Generalization

6.1 Domain Generalization

Train on multiple source domains, generalize to unseen target:

6.2 Invariant Risk Minimization (IRM)

Learn features predictive across all domains:

The penalty term encourages the classifier to be optimal in all domains simultaneously.

6.3 Distributionally Robust Optimization (DRO)

where is an uncertainty set around the training distribution.

6.4 GroupDRO

Partition data into groups, minimize worst-group risk:

where are learned group weights.

7. Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F

class GradientReversalFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, alpha):
        ctx.alpha = alpha
        return x.clone()
    
    @staticmethod
    def backward(ctx, grad_output):
        return -ctx.alpha * grad_output, None

class DANN(nn.Module):
    def __init__(self, feature_extractor, num_classes, alpha=1.0):
        super().__init__()
        self.feature_extractor = feature_extractor
        self.label_classifier = nn.Linear(feature_extractor.out_dim, num_classes)
        self.domain_classifier = nn.Sequential(
            GradientReversalFunction.apply,
            nn.Linear(feature_extractor.out_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 2),
        )
        self.alpha = alpha
    
    def forward(self, x):
        features = self.feature_extractor(x)
        class_output = self.label_classifier(features)
        domain_output = self.domain_classifier(features, self.alpha)
        return class_output, domain_output

def coral_loss(source_features, target_features):
    d = source_features.shape[1]
    source_cov = compute_cov(source_features)
    target_cov = compute_cov(target_features)
    return (source_cov - target_cov).pow(2).sum() / (4 * d ** 2)

def compute_cov(features):
    n = features.shape[0]
    mean = features.mean(dim=0, keepdim=True)
    features_centered = features - mean
    return features_centered.T @ features_centered / (n - 1)

class MMDLoss(nn.Module):
    def __init__(self, kernel='rbf', sigma=1.0):
        super().__init__()
        self.kernel = kernel
        self.sigma = sigma
    
    def forward(self, source, target):
        if self.kernel == 'rbf':
            source_source = self.rbf_kernel(source, source)
            target_target = self.rbf_kernel(target, target)
            source_target = self.rbf_kernel(source, target)
        else:
            source_source = source @ source.T
            target_target = target @ target.T
            source_target = source @ target.T
        
        n_s = source.shape[0]
        n_t = target.shape[0]
        
        mmd = (source_source.sum() / (n_s * n_s) 
               - 2 * source_target.sum() / (n_s * n_t) 
               + target_target.sum() / (n_t * n_t))
        return mmd
    
    def rbf_kernel(self, x, y):
        dist = torch.cdist(x, y, p=2)
        return torch.exp(-dist ** 2 / (2 * self.sigma ** 2))

8. SVG: Source/Target Distribution Shift

9. SVG: DANN Architecture

10. Comparison of Methods

11. Open Problems

• Multi-source adaptation: Adapting from multiple source domains
• Universal domain adaptation: Adapt without knowing label set overlap
• Test-time adaptation: Adapt during inference with batch norm
• Open-world DA: Handling unknown target classes
• Causal domain adaptation: Leveraging causal structure

References

1. Ben-David, S., et al. (2010). A Theory of Learning from Different Domains. Machine Learning.
2. Ganin, Y., & Lempitsky, V. (2015). Unsupervised Domain Adaptation by Backpropagation. ICML.
3. Long, M., et al. (2015). Learning Transferable Features with Deep Adaptation Networks. ICML.
4. Sun, B., & Saenko, K. (2016). Deep CORAL: Correlation Alignment for Deep Domain Adaptation. ECCV.
5. Arjovsky, M., et al. (2019). Invariant Risk Minimization. arXiv.