Bayesian Deep Learning: Uncertainty Quantification

Module: Machine Learning | Difficulty: Advanced

Bayesian Neural Network

Variational Inference

MC Dropout

Calibration

import numpy as np
import torch
import torch.nn as nn

class MCDropout(nn.Module):
    def __init__(self, model, n_samples=100):
        super().__init__()
        self.model = model
        self.n_samples = n_samples
    def forward(self, x):
        predictions = []
        self.model.train()  # Keep dropout on
        for _ in range(self.n_samples):
            predictions.append(self.model(x))
        predictions = torch.stack(predictions)
        mean = predictions.mean(0)
        var = predictions.var(0)
        return mean, var

| Method | ECE | NLL | Computational Cost | |--------|-----|-----|-------------------| | Standard | 0.15 | -1.2 | 1x | | MC Dropout | 0.08 | -1.5 | 100x | | Deep Ensemble | 0.05 | -1.8 | 50x | | SWAG | 0.06 | -1.7 | 2x |

Research Insight: MC dropout provides a free uncertainty estimate by running multiple forward passes with dropout enabled. The key insight is that dropout approximates a variational distribution over weights, connecting dropout to Bayesian inference.