Cross-Validation: Theory and Optimality
Module: Machine Learning | Difficulty: Advanced
k-Fold CV Risk
Leave-One-Out (LOO)
GCV (Generalized Cross-Validation)
Bias-Variance of CV
| Method | Bias | Computation | Recommended |
|---|---|---|---|
| LOO | Low | models | Small n |
| 5-Fold | Medium | 5 models | Medium n |
| 10-Fold | Low-Medium | 10 models | Large n |
| .632+ | Low | 1 model + bootstrap | Very large n |
import numpy as np
from sklearn.model_selection import KFold
def cross_validate(model, X, y, k=10, scoring='mse'):
kf = KFold(n_splits=k, shuffle=True, random_state=42)
scores = []
for train_idx, val_idx in kf.split(X):
model.fit(X[train_idx], y[train_idx])
pred = model.predict(X[val_idx])
scores.append(((y[val_idx] - pred)**2).mean())
return np.mean(scores), np.std(scores)
Research Insight: 10-fold CV has been shown to have the lowest MSE as an estimator of true prediction error, balancing bias (from fold size) and variance (from number of folds).