🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
Search courses…
💼 Servicesℹ️ About✉️ ContactView Pricing Plansfrom $10

Model Evaluation — Metrics, Cross-Validation and Selection

ML FoundationsEvaluation🟢 Free Lesson

Advertisement

ML Foundations

How to Know If Your Model Actually Works — Beyond Accuracy

Choosing the right metric and evaluation strategy is critical. A model with 99% accuracy might be useless if the data is imbalanced.

  • Precision and Recall — When false positives and false negatives matter differently
  • Cross-Validation — Getting reliable performance estimates
  • Bias-Variance Tradeoff — The central challenge in machine learning

"Not everything that counts can be counted, and not everything that can be counted counts."

Model Evaluation — Complete Guide

Choosing the right metric and evaluation strategy is critical. A model with 99% accuracy might be useless if the data is imbalanced.


Classification Metrics

Confusion Matrix and Classification MetricsConfusion MatrixPredicted →Actual →NegPosNegPos950TN50FP30FN970TPAccuracy = (950+970)/2000 = 96%Key MetricsAccuracy = (TP + TN) / TotalCorrect predictions / all predictions. Misleading for imbalanced data.Precision = TP / (TP + FP)Of predicted positives, how many are correct? When FP is costly (spam filter).Recall = TP / (TP + FN)Of actual positives, how many did we find? When FN is costly (disease detection).F1 = 2 × Precision × Recall / (Precision + Recall)

ROC Curve and AUC

ROC Curve and AUCFalse Positive RateTrue Positive Rate (Recall)0101Random (AUC=0.5)Perfect (AUC=1.0)AUC = 0.95AUC = 0.75AUC InterpretationAUC = 1.0: PerfectSeparates classes perfectlyAUC ≈¥ 0.9: ExcellentStrong discriminationAUC ≈¥ 0.7: AcceptableSome discrimination abilityAUC ≈¤ 0.5: RandomNo discrimination (useless model)Threshold Trade-offLow threshold → high recall, low precisionHigh threshold → high precision, low recall

Cross-Validation

5-Fold Cross-ValidationFold1TestTrainTrainTrainTrain→ Score₁2TrainTestTrainTrainTrain→ Score₂3-5Each fold takes a turn as test set...→ Score₃,â‚„,₄CV Score = (Score₁ + Score₂ + Score₃ + Scoreâ‚„ + Score₄) / 5Why Cross-Validation?1. Every sample used for both train and test2. Reduces variance of performance estimate3. Detects overfitting (large train/test gap)4. Better model selection and comparison5. K=5 or K=10 is standard (bias-variance tradeoff)6. Leave-One-Out: K=N, for very small datasetsMore folds = less bias, more variance (and more compute)

Bias-Variance Tradeoff

Bias-Variance Tradeoff: The Central ChallengeModel Complexity →ErrorTrain errorTest errorBias²VarianceSweet SpotUnderfittingHigh Bias, Low VarianceOverfittingLow Bias, High VarianceTraining errorTest error (generalization)Optimal complexity

Regression Metrics Comparison


Key Takeaways


What to Learn Next

-> Regularization Prevent overfitting with Ridge, Lasso, and Elastic Net.

-> Model Selection Hyperparameter tuning, grid search, and choosing the best model.

-> Ensemble Methods Bagging, boosting, and stacking for stronger models.

Need Expert Machine Learning Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement