ROC Curves and AUC � Model Discrimination
Statistics
Evaluating How Well Classifiers Separate Classes
ROC curves plot true positive rates against false positive rates across all thresholds, while AUC summarizes overall discrimination ability. These threshold-independent metrics reveal a models inherent ability to distinguish between classes.
- Medical Screening � Compare diagnostic tests for disease detection accuracy
- Fraud Detection � Evaluate model performance across operating thresholds
- Credit Risk � Assess borrower classification before setting cutoff policies
An AUC of 0.5 means random guessing; an AUC of 1 means perfect separation.
ROC curves and AUC measure how well a classifier distinguishes between classes. They are threshold-independent metrics that evaluate the discrimination ability of a model.
Key Metrics
Area Under the Curve (AIC)
| AUC | Interpretation |
|---|---|
| 0.5 | No discrimination (random guessing) |
| 0.5 - 0.7 | Poor discrimination |
| 0.7 - 0.8 | Acceptable discrimination |
| 0.8 - 0.9 | Excellent discrimination |
| > 0.9 | Outstanding discrimination |
Threshold Selection
The optimal threshold depends on the cost ratio of false positives vs false negatives.
Confusion Matrix
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | TP | FN |
| Actual Negative | FP | TN |
Precision-Recall Curve
When classes are imbalanced, the PR curve may be more informative than ROC.
Multi-Class Extensions
One-vs-Rest (OvR)
Compute ROC for each class against all others, then average.
One-vs-One (OvO)
Compute ROC for each pair of classes.
DeLong Test
Tests whether two AUCs are significantly different.
Python Implementation
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (roc_curve, roc_auc_score, precision_recall_curve,
confusion_matrix, classification_report)
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
np.random.seed(42)
# Generate data
X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
weights=[0.7], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Fit model
model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)
y_prob = model.predict_proba(X_test)[:, 1]
# ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
auc = roc_auc_score(y_test, y_prob)
# Find optimal threshold (Youden's J)
J = tpr - fpr
optimal_idx = np.argmax(J)
optimal_threshold = thresholds[optimal_idx]
# Plot
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
# ROC
axes[0].plot(fpr, tpr, 'b-', label=f'AUC = {auc:.3f}')
axes[0].plot([0, 1], [0, 1], 'r--', label='Random')
axes[0].scatter(fpr[optimal_idx], tpr[optimal_idx], c='red', s=100, label=f'Threshold = {optimal_threshold:.2f}')
axes[0].set_xlabel('FPR')
axes[0].set_ylabel('TPR')
axes[0].set_title('ROC Curve')
axes[0].legend()
# Precision-Recall
precision, recall, _ = precision_recall_curve(y_test, y_prob)
axes[1].plot(recall, precision, 'b-')
axes[1].set_xlabel('Recall')
axes[1].set_ylabel('Precision')
axes[1].set_title('Precision-Recall Curve')
plt.tight_layout()
plt.show()
# Confusion matrix at optimal threshold
y_pred_optimal = (y_prob >= optimal_threshold).astype(int)
print(f"Optimal threshold: {optimal_threshold:.3f}")
print(f"Confusion Matrix:\n{confusion_matrix(y_test, y_pred_optimal)}")
print(f"\n{classification_report(y_test, y_pred_optimal)}")
Worked Example
Key Takeaways
Related Topics
- See Cross-Validation for model evaluation methodology
- See AIC and BIC for model selection criteria
- See Missing Data for data quality considerations