Cross-Validation in Statistics

Statistics

Estimating How Well Models Generalize to New Data

Cross-validation partitions data into training and validation sets, repeatedly fitting and evaluating models to estimate out-of-sample performance. It prevents overfitting by testing models on data they haven't seen during training.

Model Selection — Choose between competing models with honest performance estimates
Hyperparameter Tuning — Find optimal settings without overfitting to validation data
Clinical Prediction — Validate risk scores on held-out patient populations

Cross-validation is the closest thing to a crystal ball for predicting model performance.

Cross-validation (CV) estimates how well a model generalizes to unseen data by training and testing on different subsets of the available data.

Why Cross-Validation?

K-Fold Cross-Validation

The most common CV method. Data is split into k roughly equal folds.

Steps

| Step | Action |

|------|--------|

| 1 | Randomly partition data into k folds |

| 2 | For each fold i: train on k-1 folds, test on fold i |

| 3 | Compute the error metric for each fold |

| 4 | Average the k error estimates |

Common Values of k

|---|------|------|----------|------|

Leave-One-Out Cross-Validation

Each observation serves as the test set exactly once.

Stratified Cross-Validation

Ensures each fold has approximately the same class proportions as the full dataset.

Repeated Cross-Validation

Repeat k-fold CV multiple times with different random partitions to reduce variance.

| Repetition | Description |

|-----------|-------------|

| 1 × 10-CV | Standard 10-fold |

| 5 × 2-CV | 5 repetitions of 2-fold |

| 10 × 10-CV | 10 repetitions of 10-fold |

Nested Cross-Validation

For simultaneous model selection and performance estimation.

| Loop | Purpose |

|------|---------|

| Outer | Test on held-out fold -> unbiased performance estimate |

| Inner | Tune hyperparameters on training fold -> model selection |

Python Implementation


import numpy as np

import pandas as pd

from sklearn.model_selection import (KFold, LeaveOneOut, cross_val_score,

                                      StratifiedKFold, RepeatedKFold, cross_val_predict)

from sklearn.linear_model import LinearRegression, Ridge

from sklearn.datasets import make_regression

from sklearn.metrics import mean_squared_error

import matplotlib.pyplot as plt



np.random.seed(42)



# Generate data

X, y = make_regression(n_samples=200, n_features=5, noise=10, random_state=42)



# 10-Fold CV

kf = KFold(n_splits=10, shuffle=True, random_state=42)

lr = LinearRegression()

scores_10fold = cross_val_score(lr, X, y, cv=kf, scoring='neg_mean_squared_error')

print(f"10-Fold CV MSE: {-scores_10fold.mean():.2f} (+/- {scores_10fold.std():.2f})")



# LOO-CV

loo = LeaveOneOut()

scores_loo = cross_val_score(lr, X, y, cv=loo, scoring='neg_mean_squared_error')

print(f"LOO-CV MSE: {-scores_loo.mean():.2f}")



# Repeated 5-Fold CV

rkf = RepeatedKFold(n_splits=5, n_repeats=10, random_state=42)

scores_repeated = cross_val_score(lr, X, y, cv=rkf, scoring='neg_mean_squared_error')

print(f"Repeated 5-Fold MSE: {-scores_repeated.mean():.2f} (+/- {scores_repeated.std():.2f})")



# Nested CV for model selection

alphas = [0.1, 1.0, 10.0, 100.0]

outer_scores = []

for train_idx, test_idx in KFold(n_splits=5, shuffle=True, random_state=42).split(X):

    X_train, X_test = X[train_idx], X[test_idx]

    y_train, y_test = y[train_idx], y[test_idx]

    

    # Inner loop: select alpha

    best_alpha = None

    best_score = -np.inf

    for alpha in alphas:

        ridge = Ridge(alpha=alpha)

        inner_scores = cross_val_score(ridge, X_train, y_train, cv=3, scoring='neg_mean_squared_error')

        if inner_scores.mean() > best_score:

            best_score = inner_scores.mean()

            best_alpha = alpha

    

    # Outer loop: evaluate

    ridge_best = Ridge(alpha=best_alpha)

    ridge_best.fit(X_train, y_train)

    outer_scores.append(mean_squared_error(y_test, ridge_best.predict(X_test)))

    

print(f"\nNested CV MSE: {np.mean(outer_scores):.2f} (+/- {np.std(outer_scores):.2f})")

print(f"Selected alpha: {best_alpha}")

Cross-Validation in Statistics

Cross-Validation in Statistics

Estimating How Well Models Generalize to New Data

Why Cross-Validation?

K-Fold Cross-Validation

Steps

Common Values of k

Leave-One-Out Cross-Validation

Stratified Cross-Validation

Repeated Cross-Validation

Nested Cross-Validation

Python Implementation

Worked Example

Key Takeaways

Related Topics

Need Expert Statistics Help?