Model Evaluation: ROC, AUC, Precision, Recall and F1

Why Model Evaluation Matters

A model is only as good as our ability to measure its performance. Choosing the wrong metric can lead to catastrophically poor deployment decisions. This lesson covers the essential evaluation toolkit every ML practitioner must master.

1. The Confusion Matrix

For a binary classifier, the confusion matrix partitions predictions into four categories:

Key Definitions

• Positive class = the class of interest (e.g., "has cancer", "fraudulent transaction")
• TP: Model predicts positive, actual is positive
• FP: Model predicts positive, actual is negative (Type I error)
• FN: Model predicts negative, actual is positive (Type II error)
• TN: Model predicts negative, actual is negative

2. Classification Metrics

2.1 Accuracy

The fraction of all predictions that are correct. Dangerous when classes are imbalanced. A model predicting "no cancer" for all patients in a population with 1% prevalence achieves 99% accuracy while being completely useless.

2.2 Precision (Positive Predictive Value)

Of all instances the model predicted as positive, what fraction actually are positive? High precision means few false alarms.

Use when: False positives are costly (e.g., spam filter marking legitimate emails as spam).

2.3 Recall (Sensitivity, True Positive Rate)

Of all actual positive instances, what fraction did the model catch? High recall means few missed positives.

Use when: False negatives are costly (e.g., failing to detect cancer).

2.4 F1-Score

The harmonic mean of precision and recall. Balances both concerns. The harmonic mean penalizes extreme imbalances more than the arithmetic mean.

2.5 Fβ-Score (Generalized)

• $\beta = 1$: Equal weight (standard F1)
• $\beta = 2$: Recall weighted 2× more than precision
• $\beta = 0.5$: Precision weighted 2× more than recall

Comparison Table

3. The ROC Curve

3.1 Concept

The Receiver Operating Characteristic (ROC) curve plots the tradeoff between catching positives and raising false alarms as the classification threshold varies.

For each possible threshold $t \in [0, 1]$:

1. Classify instances with score ≥ $t$ as positive
2. Compute (FPR, TPR) pair
3. Plot the curve

3.2 Interpreting the ROC Curve

• Top-left corner (0, 1): Perfect classifier — catches all positives with no false alarms
• Diagonal line: Random guessing — no discriminative power
• Below diagonal: Worse than random (usually indicates flipped labels)
• Closer to top-left: Better model

3.3 Threshold Selection

The ROC curve does not tell you which threshold to use. Common strategies:

1. Closest to (0, 1): Minimize $\sqrt{\text{FPR}^2 + (1 - \text{TPR})^2}$
2. Youden's J statistic: Maximize $J = \text{TPR} - \text{FPR}$
3. Cost-based: Assign costs to FP and FN, minimize expected cost

4. AUC — Area Under the ROC Curve

Equivalently, AUC is the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance:

AUC Interpretation

AUC is threshold-invariant: It evaluates the model's ranking ability across all thresholds simultaneously.

AUC is class-prevalence-invariant: Unlike accuracy, it doesn't change when class proportions shift.

5. Precision-Recall (PR) Curve

When the positive class is rare, the PR curve provides a more informative view than ROC.

The PR curve plots Precision (y-axis) vs. Recall (x-axis) across all thresholds.

Average Precision (AP)

where $R_k$ and $P_k$ are recall and precision at the $k$-th threshold. AP summarizes the PR curve into a single number (area under the PR curve).

When to Use PR vs. ROC

6. Regression Metrics

6.1 Mean Squared Error (MSE)

Penalizes large errors quadratically. Sensitive to outliers. Units are squared.

6.2 Root Mean Squared Error (RMSE)

Same units as the target variable. More interpretable than MSE.

6.3 Mean Absolute Error (MAE)

Robust to outliers. Linear penalty for errors.

6.4 R² (Coefficient of Determination)

• $R^2 = 1$: Perfect prediction
• $R^2 = 0$: Model performs no better than predicting the mean
• $R^2 < 0$: Model is worse than the mean

Metric Comparison

7. Bias-Variance Tradeoff

The expected prediction error for a model can be decomposed as:

Definitions

Bias (systematic error): Error from erroneous assumptions in the learning algorithm. High bias causes the model to miss relevant relations between features and targets.

Variance (sensitivity to data): Error from sensitivity to small fluctuations in the training set. High variance causes the model to model random noise.

The Tradeoff

• Increasing model complexity → decreases bias, increases variance
• Decreasing model complexity → increases bias, decreases variance
• Optimal complexity → minimizes total error

Practical Implications

8. Learning Curves

Learning curves plot model performance as a function of training set size, revealing whether the model suffers from bias, variance, or both.

Reading Learning Curves

High Variance (overfitting):

• Training error: Low and stays low
• Validation error: Starts high, slowly decreases
• Large gap between curves
• Fix: More data, regularization, simpler model, feature reduction

High Bias (underfitting):

• Training error: Starts moderate, stays moderate/high
• Validation error: Starts high, converges toward training error
• Small gap, both converge to high error
• Fix: More complex model, more features, reduce regularization

The Irreducible Error

Both curves converge to a floor above zero — this is the irreducible error ($\sigma^2_\epsilon$), the noise inherent in the data that no model can eliminate.

9. Summary Table

10. Key Takeaways

1. Never use accuracy alone with imbalanced classes — use precision, recall, F1, or AUC.
2. ROC curves show the full threshold tradeoff; AUC summarizes ranking ability.
3. PR curves are more informative than ROC when the positive class is rare.
4. Bias-variance decomposition diagnoses underfitting vs. overfitting.
5. Learning curves reveal whether more data or a more complex model will help.
6. Choose metrics aligned with business costs — what matters more: false alarms or missed detections?