Bayesian Model Selection: Marginal Likelihood and BIC

Module: Machine Learning | Difficulty: Advanced

Marginal Likelihood

BIC

AIC

Bayes Factor

Occam's Razor

The marginal likelihood automatically penalizes model complexity.

import numpy as np
from scipy.integrate import quad

def bayes_factor(data, model1, model2, prior1, prior2):
    def integrand1(theta):
        return np.exp(model1.log_likelihood(data, theta)) * prior1(theta)
    def integrand2(theta):
        return np.exp(model2.log_likelihood(data, theta)) * prior2(theta)
    evidence1, _ = quad(integrand1, -np.inf, np.inf)
    evidence2, _ = quad(integrand2, -np.inf, np.inf)
    return evidence1 / evidence2

Research Insight: The marginal likelihood is the gold standard for model comparison because it automatically implements Occam's razor. Complex models are penalized because they spread their probability mass over more parameter space.