Why Cross-Validation?
The Holdout Method Limitation
The naive holdout approach splits data into training and test sets once. This has critical flaws:
Key limitations:
- Performance estimate has high variance (depends on single split)
- Wastes data (test set never used for training)
- Can't assess model stability
- Risk of optimistic/pessimistic bias
K-Fold Cross-Validation
The gold standard for model evaluation. Split data into folds, train on , test on 1, rotate.
Mathematical Formulation
For dataset partitioned into folds :
where is the model trained on all data except fold , and is the loss function.
Choice of k
| k | Pros | Cons |
|---|---|---|
| 5 | Good bias-variance tradeoff | Standard choice |
| 10 | Lower bias estimate | Higher computational cost |
| (LOO) | Nearly unbiased | High variance, expensive |
Stratified K-Fold
Ensures each fold maintains the same class distribution as the full dataset critical for imbalanced problems.
Leave-One-Out (LOO) Cross-Validation
A special case of K-Fold where (number of samples):
Characteristics:
- Nearly unbiased estimate of generalization error
- High variance (each training set differs by only 1 sample)
- Computationally expensive: model fits
- Approximately equivalent to AIC for linear models
Time Series Cross-Validation
Standard K-Fold violates temporal ordering. Use expanding or sliding windows instead.
Bias-Variance Tradeoff
Mathematical Decomposition
For model trained on dataset , the expected prediction error at point decomposes as:
where:
Intuition
Underfitting vs Overfitting Diagnosis
Diagnostic Summary
| Symptom | Diagnosis | Remedy |
|---|---|---|
| Train acc ∫ Val acc | Overfitting | Regularization, more data, simpler model |
| Train acc ∇ Val acc (both low) | Underfitting | More features, complex model |
| High CV variance | Unstable model | More data, simpler model, ensemble |
Model Selection with Cross-Validation
Use nested cross-validation to avoid optimistic bias when selecting hyperparameters:
Implementation in Python
Key Takeaways
- Always use cross-validation holdout estimates are unreliable
- Stratified K-Fold is essential for classification (especially imbalanced)
- Time series require temporal ordering never shuffle
- Bias-variance tradeoff is fundamental: optimize the total error, not just bias
- Learning curves reveal whether you need more data, more features, or regularization
- Nested CV avoids optimistic bias in model selection
- Variance of CV scores matters high variance signals instability