Bias-Variance Tradeoff: Mathematical Analysis

Module: Machine Learning | Difficulty: Advanced

Generalization Error Decomposition

where is irreducible noise.

Bias

Variance

Model Complexity and Overfitting

| Model | Bias | Variance | Risk | |-------|------|----------|------| | Linear Regression | High | Low | | | Decision Tree | Low | High | | | Random Forest | Low | Medium | | | Neural Network | Very Low | Very High | |

Estimating Bias and Variance

import numpy as np
from sklearn.model_selection import KFold

def estimate_bias_variance(model, X, y, n_bootstrap=100):
    predictions = np.zeros((n_bootstrap, len(y)))
    for i in range(n_bootstrap):
        idx = np.random.choice(len(y), len(y), replace=True)
        model.fit(X[idx], y[idx])
        predictions[i] = model.predict(X)
    bias_sq = (predictions.mean(axis=0) - y)**2
    variance = predictions.var(axis=0)
    return bias_sq.mean(), variance.mean()

Research Insight: The bias-variance tradeoff is not always monotone. Double descent shows that very large models can have both low bias AND low variance simultaneously.