How Good Is Your Model? — The Complete Evaluation Toolkit
Model evaluation separates good predictions from lucky guesses. Master cross-validation, ROC curves, bias-variance tradeoff, and hyperparameter tuning.
- Cross-Validation — Reliable performance estimation
- Confusion Matrix — Where your model gets confused
- ROC & AUC — Threshold-independent evaluation
"Without data, you're just another person with an opinion." — W. Edwards Deming
Prerequisites
Before diving in, make sure you're comfortable with:
- Basic Probability — Conditional probability, Bayes theorem
- Classification — What predictions, true/false positives/negatives are
- Python — NumPy, scikit-learn basics
- Statistics — Mean, variance, standard deviation
Learning Objectives
After completing this tutorial, you will be able to:
- Compute and interpret confusion matrix, accuracy, precision, recall, F1
- Implement k-fold cross-validation and understand stratified splits
- Plot and interpret ROC curves and compute AUC
- Explain the bias-variance tradeoff and its practical implications
- Apply hyperparameter tuning strategies (GridSearch, RandomSearch, Bayesian)
- Know which metric to use for different problem types
Model Evaluation — Complete Guide
Evaluating model performance is the most critical step in the ML pipeline — a model that doesn't generalize is useless.
The Fundamental Problem
Classification Metrics
Confusion Matrix
Precision, Recall, F1
from sklearn.metrics import (confusion_matrix, accuracy_score, precision_score,
recall_score, f1_score, classification_report)
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2, random_state=42)
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
print(f"\nAccuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"Precision: {precision_score(y_test, y_pred):.3f}")
print(f"Recall: {recall_score(y_test, y_pred):.3f}")
print(f"F1 Score: {f1_score(y_test, y_pred):.3f}")
print(f"\n{classification_report(y_test, y_pred, target_names=data.target_names)}")
ROC Curve & AUC
from sklearn.metrics import roc_curve, auc, roc_auc_score
import matplotlib.pyplot as plt
y_prob = model.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
roc_auc = auc(fpr, tpr)
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, 'b-', linewidth=2, label=f'Random Forest (AUC = {roc_auc:.3f})')
plt.plot([0, 1], [0, 1], 'k--', linewidth=1, label='Random (AUC = 0.5)')
plt.fill_between(fpr, tpr, alpha=0.2, color='blue')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve — Breast Cancer Classification')
plt.legend(loc='lower right')
plt.grid(True, alpha=0.3)
plt.savefig("roc_curve.png", dpi=150)
plt.show()
print(f"AUC Score: {roc_auc_score(y_test, y_prob):.3f}")
Cross-Validation
from sklearn.model_selection import cross_val_score, StratifiedKFold
import numpy as np
# Standard k-fold
scores = cross_val_score(model, data.data, data.target, cv=5, scoring='accuracy')
print(f"5-Fold CV: {scores.mean():.3f} ± {scores.std():.3f}")
# Stratified k-fold (preserves class proportions)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
stratified_scores = cross_val_score(model, data.data, data.target, cv=skf, scoring='f1')
print(f"Stratified F1: {stratified_scores.mean():.3f} ± {stratified_scores.std():.3f}")
# Cross-validation with multiple metrics
from sklearn.model_selection import cross_validate
results = cross_validate(model, data.data, data.target, cv=5,
scoring=['accuracy', 'f1', 'roc_auc'])
print(f"\nAccuracy: {results['test_accuracy'].mean():.3f}")
print(f"F1: {results['test_f1'].mean():.3f}")
print(f"AUC: {results['test_roc_auc'].mean():.3f}")
Bias-Variance Tradeoff
from sklearn.model_selection import learning_curve
import matplotlib.pyplot as plt
train_sizes, train_scores, val_scores = learning_curve(
RandomForestClassifier(n_estimators=100, random_state=42),
data.data, data.target, cv=5,
train_sizes=np.linspace(0.1, 1.0, 10), scoring='accuracy', n_jobs=-1)
plt.figure(figsize=(8, 5))
plt.plot(train_sizes, train_scores.mean(axis=1), 'o-', label='Training score')
plt.plot(train_sizes, val_scores.mean(axis=1), 'o-', label='Cross-validation score')
plt.fill_between(train_sizes, train_scores.mean(axis=1) - train_scores.std(axis=1),
train_scores.mean(axis=1) + train_scores.std(axis=1), alpha=0.2)
plt.fill_between(train_sizes, val_scores.mean(axis=1) - val_scores.std(axis=1),
val_scores.mean(axis=1) + val_scores.std(axis=1), alpha=0.2)
plt.xlabel('Training Set Size')
plt.ylabel('Accuracy')
plt.title('Learning Curve — Diagnose Bias vs Variance')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig("learning_curve.png", dpi=150)
plt.show()
Hyperparameter Tuning
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from scipy.stats import randint, uniform
# Grid Search
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [3, 5, 10, None],
'min_samples_split': [2, 5, 10],
}
grid_search = GridSearchCV(RandomForestClassifier(random_state=42),
param_grid, cv=5, scoring='accuracy', n_jobs=-1, verbose=1)
grid_search.fit(X_train, y_train)
print(f"Grid Search best: {grid_search.best_params_}")
# Random Search (more efficient)
param_distributions = {
'n_estimators': randint(50, 300),
'max_depth': randint(3, 20),
'min_samples_split': randint(2, 20),
'min_samples_leaf': randint(1, 10),
'max_features': uniform(0.1, 0.9),
}
random_search = RandomizedSearchCV(RandomForestClassifier(random_state=42),
param_distributions, n_iter=50, cv=5,
scoring='accuracy', n_jobs=-1, random_state=42)
random_search.fit(X_train, y_train)
print(f"Random Search best: {random_search.best_params_}")
Regression Metrics
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np
y_true = np.array([3, -0.5, 2, 7])
y_pred = np.array([2.5, 0.0, 2, 8])
print(f"MSE: {mean_squared_error(y_true, y_pred):.3f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_true, y_pred)):.3f}")
print(f"MAE: {mean_absolute_error(y_true, y_pred):.3f}")
print(f"R²: {r2_score(y_true, y_pred):.3f}")
Real-World Applications
Medical Diagnosis
Metrics matter enormously: high recall ensures we catch all cancer cases (FN are fatal), while high precision avoids unnecessary biopsies (FP cause anxiety and cost). F1 or AUC-ROC balance these concerns.
Credit Scoring
AUC-ROC is ideal because it evaluates across all thresholds — lenders can adjust the decision threshold based on risk tolerance. Cross-validation prevents overfitting to historical data.
Spam Filtering
Precision is king — misclassifying legitimate email as spam (FP) is much worse than letting spam through (FN). Use precision@k: "of the top k emails flagged, how many are actually spam?"
Manufacturing Quality Control
Recall is critical — missing a defective product (FN) can cause recalls and lawsuits. Accept more false positives (FP) that trigger manual inspection rather than let defects slip through.
Recommendation Systems
Precision@k and Recall@k evaluate top-k recommendations. NDCG@k accounts for ranking quality — relevant items should appear higher in the list.
Common Mistakes & How to Avoid Them
Mistake 1: Using accuracy on imbalanced data
- Problem: 95% accuracy is useless if 95% of data is one class
- Solution: Use F1, AUC-ROC, or precision/recall for imbalanced problems
Mistake 2: Not using cross-validation
- Problem: Single train/test split gives high-variance estimate
- Solution: Always use k-fold CV (k=5 or 10) for reliable performance estimates
Mistake 3: Tuning on test set
- Problem: Using test set for hyperparameter selection → overfitting to test set
- Solution: Split into train/validation/test; tune on validation, final evaluation on test
Mistake 4: Ignoring data leakage
- Problem: Preprocessing on full dataset before splitting leaks information
- Solution: Use
Pipeline— fit only on training data, transform both train and test
Mistake 5: Choosing wrong metric for the problem
- Problem: Maximizing accuracy when recall matters (cancer detection)
- Solution: Match metric to business cost: FP cost vs FN cost → choose precision or recall
Mistake 6: Not stratifying splits
- Problem: Random splits may create imbalanced class distributions in folds
- Solution: Use
StratifiedKFoldfor classification tasks
Interview Questions
Q1: What is the difference between precision and recall? A: Precision = TP/(TP+FP) — "Of all predicted positives, how many are correct?" Recall = TP/(TP+FN) — "Of all actual positives, how many did we catch?" Trade-off: lowering threshold increases recall but decreases precision.
Q2: When would you use AUC-ROC vs F1 score? A: AUC-ROC: threshold-independent, good for comparing models, works well with balanced data. F1: single-threshold metric, better for imbalanced data, more interpretable for stakeholders. Use both when possible.
Q3: Explain the bias-variance tradeoff. A: Bias = error from wrong model assumptions (underfitting). Variance = error from sensitivity to training data (overfitting). Total error = Bias² + Variance + irreducible noise. Simple models: high bias, low variance. Complex models: low bias, high variance. Goal: find the sweet spot.
Q4: Why use cross-validation instead of a single train/test split? A: Single split gives high-variance estimate that depends on which points end up in test set. CV averages over K estimates, reducing variance. Also uses all data for both training and evaluation (at different times).
Q5: What is data leakage and how do you prevent it? A: Leakage = test set information influencing training. Examples: scaling before splitting, feature selection on full data, temporal leakage (future data in past training). Prevention: always split first, use Pipeline, never fit preprocessing on test data.
Q6: How do you handle class imbalance in evaluation? A: Use appropriate metrics (F1, AUC-ROC, precision, recall) instead of accuracy. Use stratified splits to maintain class proportions. Consider class_weight='balanced' in models. For severe imbalance, use SMOTE or undersampling (but only on training data).
Q7: When would you prefer precision over recall? A: When false positives are costly: spam filter (deleting good email), fraud detection (blocking legitimate transactions), content moderation (removing acceptable content). High precision = fewer false alarms.
Practice Exercise
Challenge: Complete Model Evaluation Pipeline
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_cross_val, cross_val_predict, GridSearchCV
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import (confusion_matrix, classification_report, roc_curve, auc,
precision_recall_curve, average_precision_score)
import matplotlib.pyplot as plt
import numpy as np
# Create imbalanced dataset
X, y = make_classification(n_samples=1000, n_features=20, weights=[0.9, 0.1],
flip_y=0.05, random_state=42)
# Split: train/val/test = 60/20/20
X_trainval, X_test, y_trainval, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_trainval, y_trainval, test_size=0.25, random_state=42, stratify=y_trainval)
# Compare models with cross-validation
models = {
'Logistic Regression': Pipeline([('scaler', StandardScaler()), ('clf', LogisticRegression())]),
'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
'Gradient Boosting': GradientBoostingClassifier(n_estimators=100, random_state=42),
}
print("Cross-Validation Results (F1):")
print("-" * 50)
for name, model in models.items():
scores = cross_val_predict(model, X_train, y_train, cv=5, method='predict')
from sklearn.metrics import f1_score
print(f"{name:25s} F1={f1_score(y_train, scores):.3f}")
# Hyperparameter tuning on validation set
param_grid = {'n_estimators': [50, 100, 200], 'max_depth': [3, 5, 10]}
grid = GridSearchCV(RandomForestClassifier(random_state=42), param_grid, cv=5, scoring='f1')
grid.fit(X_train, y_train)
print(f"\nBest params: {grid.best_params_}")
# Final evaluation on test set
y_pred = grid.predict(X_test)
print(f"\nTest Set Classification Report:")
print(classification_report(y_test, y_pred))
# Plot ROC and Precision-Recall curves
y_prob = grid.predict_proba(X_test)[:, 1]
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
fpr, tpr, _ = roc_curve(y_test, y_prob)
ax1.plot(fpr, tpr, linewidth=2, label=f'AUC = {auc(fpr, tpr):.3f}')
ax1.plot([0, 1], [0, 1], 'k--')
ax1.set_xlabel('FPR')
ax1.set_ylabel('TPR')
ax1.set_title('ROC Curve')
ax1.legend()
precision, recall, _ = precision_recall_curve(y_test, y_prob)
ax2.plot(recall, precision, linewidth=2, label=f'AP = {average_precision_score(y_test, y_prob):.3f}')
ax2.set_xlabel('Recall')
ax2.set_ylabel('Precision')
ax2.set_title('Precision-Recall Curve')
ax2.legend()
plt.tight_layout()
plt.savefig("evaluation_curves.png", dpi=150)
plt.show()
Comparison Table
Metrics at a Glance
| Metric | Best For | Weakness | Range |
|---|---|---|---|
| Accuracy | Balanced classes | Misleading with imbalance | [0, 1] |
| Precision | Low FP important | Ignores FN | [0, 1] |
| Recall | Low FN important | Ignores FP | [0, 1] |
| F1 Score | Balance Prec/Rec | Harder to interpret | [0, 1] |
| AUC-ROC | Threshold-independent | Less intuitive for stakeholders | [0, 1] |
| MSE/RMSE | Regression, large errors | Sensitive to outliers | [0, ∞) |
| R² | Variance explained | Can be negative | (-∞, 1] |
Key Formulas Reference
| Formula | Expression | Context |
|---|---|---|
| Accuracy | Overall correctness | |
| Precision | Positive predictive value | |
| Recall | Sensitivity, true positive rate | |
| F1 | Harmonic mean | |
| AUC-ROC | Threshold-independent | |
| Bias-Variance | Error decomposition |
Key Takeaways
What to Learn Next
-> Feature Engineering Better features often matter more than model choice.
-> Hyperparameter Optimization Advanced tuning: Bayesian optimization, Optuna, Hyperband.
-> ML Pipelines Production-ready workflows with scikit-learn Pipeline and deployment.