ROC Curves and AUC � Model Discrimination

ROC curves and AUC measure how well a classifier distinguishes between classes. They are threshold-independent metrics that evaluate the discrimination ability of a model.

Key Metrics

Area Under the Curve (AIC)

Threshold Selection

The optimal threshold depends on the cost ratio of false positives vs false negatives.

Confusion Matrix

Precision-Recall Curve

When classes are imbalanced, the PR curve may be more informative than ROC.

Multi-Class Extensions

One-vs-Rest (OvR)

Compute ROC for each class against all others, then average.

One-vs-One (OvO)

Compute ROC for each pair of classes.

DeLong Test

Tests whether two AUCs are significantly different.

Python Implementation

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (roc_curve, roc_auc_score, precision_recall_curve,
                              confusion_matrix, classification_report)
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

np.random.seed(42)

# Generate data
X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
                           weights=[0.7], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Fit model
model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)
y_prob = model.predict_proba(X_test)[:, 1]

# ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
auc = roc_auc_score(y_test, y_prob)

# Find optimal threshold (Youden's J)
J = tpr - fpr
optimal_idx = np.argmax(J)
optimal_threshold = thresholds[optimal_idx]

# Plot
fig, axes = plt.subplots(1, 2, figsize=(12, 5))

# ROC
axes[0].plot(fpr, tpr, 'b-', label=f'AUC = {auc:.3f}')
axes[0].plot([0, 1], [0, 1], 'r--', label='Random')
axes[0].scatter(fpr[optimal_idx], tpr[optimal_idx], c='red', s=100, label=f'Threshold = {optimal_threshold:.2f}')
axes[0].set_xlabel('FPR')
axes[0].set_ylabel('TPR')
axes[0].set_title('ROC Curve')
axes[0].legend()

# Precision-Recall
precision, recall, _ = precision_recall_curve(y_test, y_prob)
axes[1].plot(recall, precision, 'b-')
axes[1].set_xlabel('Recall')
axes[1].set_ylabel('Precision')
axes[1].set_title('Precision-Recall Curve')
plt.tight_layout()
plt.show()

# Confusion matrix at optimal threshold
y_pred_optimal = (y_prob >= optimal_threshold).astype(int)
print(f"Optimal threshold: {optimal_threshold:.3f}")
print(f"Confusion Matrix:\n{confusion_matrix(y_test, y_pred_optimal)}")
print(f"\n{classification_report(y_test, y_pred_optimal)}")

Worked Example

Key Takeaways

Related Topics

• See Cross-Validation for model evaluation methodology
• See AIC and BIC for model selection criteria
• See Missing Data for data quality considerations