Model Selection
Overview
The bias-variance tradeoff decomposes prediction error into biasΒ² (error from incorrect assumptions), variance (error from sensitivity to training data), and irreducible noise. Cross-validation estimates out-of-sample performance by training on folds and testing on the held-out fold, repeating for each fold. AIC () minimizes prediction error and favors larger models. BIC () penalizes complexity more heavily and favors simpler models. Regularization (Ridge/Lasso) implicitly selects complexity by shrinking coefficients β Lasso drives some to zero for automatic feature selection. The goal is always to minimize expected prediction error on new data.
Key Concepts
AIC vs BIC
| Criterion | Penalty | Favors | Best For | Consistency |
|---|---|---|---|---|
| AIC | Larger models | Prediction accuracy | No | |
| BIC | Simpler models | Interpretability | Yes |
Regularization Comparison
| Method | Penalty | Effect | Feature Selection? |
|---|---|---|---|
| Ridge (L2) | Shrinks all coefficients | No | |
| Lasso (L1) | Drives some to zero | Yes | |
| Elastic Net | L1 + L2 | Combines both | Partially |
Quick Example
Key Takeaways
Deep Dive
For detailed explanations, worked examples, and Python implementations, explore the dedicated statistics lessons:
Cross-Validation
- Cross-Validation β K-fold, stratified, leave-one-out, and nested cross-validation
Information Criteria
- AIC and BIC β Derivation, interpretation, model averaging, and when each is appropriate
ROC and AUC
- ROC and AUC β Threshold-independent evaluation, ROC curves, AUC interpretation, and trade-offs
Related Topics
- Regression Assumptions β What happens when model assumptions are violated
- Multiple Linear Regression β Variable selection in regression models
- Ridge Regression β L2 regularization for multicollinearity
- Lasso Regression β L1 regularization for feature selection
- Elastic Net β Combining L1 and L2 penalties