Classification with Probability — From Linear to Sigmoid
Logistic regression transforms linear outputs into probabilities using the sigmoid function. It is the foundation of classification in machine learning.
- Sigmoid Function — Map any real number to a probability between 0 and 1
- Cross-Entropy Loss — The cost function that powers classification training
- Multiclass Extension — Softmax regression for multiple classes
"The goal is to turn data into information, and information into insight."
Prerequisites
Before diving in, make sure you're comfortable with:
- Linear Regression — How it works, cost function, gradient descent
- Basic Calculus — Derivatives, chain rule, gradients
- Probability — Conditional probability, Bayes' theorem
- Python/NumPy — Arrays, functions, basic ML workflows
Learning Objectives
After completing this tutorial, you will be able to:
- Explain how the sigmoid function maps outputs to probabilities
- Derive and apply binary cross-entropy loss
- Implement logistic regression from scratch and with sklearn
- Understand decision boundaries and multiclass extension (softmax)
- Interpret AUC-ROC and choose appropriate classification thresholds
- Apply regularization to prevent overfitting in logistic regression
Logistic Regression — Complete Guide for Classification
Despite its name, logistic regression is a classification algorithm. It predicts the probability that an input belongs to a class.
From Linear to Logistic Regression
Cost Function
Decision Boundary
Gradient Derivation
Python Implementation
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = LogisticRegression(C=1.0, penalty='l2', solver='lbfgs')
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]
from sklearn.metrics import accuracy_score, roc_auc_score
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"AUC-ROC: {roc_auc_score(y_test, y_prob):.3f}")
Real-World Applications
Email Spam Detection
Gmail uses logistic regression (among other models) to classify emails as spam or not spam. Features include word frequencies, sender reputation, and email metadata. The probability output allows threshold tuning — catching more spam (higher recall) at the cost of more false positives.
Credit Card Fraud Detection
Banks use logistic regression to flag fraudulent transactions in real-time. The model outputs the probability that a transaction is fraudulent, and transactions above a threshold are blocked. Class imbalance is extreme (0.1% fraud), making AUC-ROC more useful than accuracy.
Medical Diagnosis
Logistic regression predicts whether a patient has a disease based on symptoms, lab results, and demographics. The interpretable coefficients show which factors most strongly predict disease — crucial for medical decision-making and regulatory approval.
Customer Churn Prediction
Telecom companies predict which customers will cancel their service. Features include usage patterns, billing history, and customer service interactions. The probability output helps prioritize retention efforts for high-risk customers.
A/B Testing Analysis
Logistic regression analyzes whether a new website design increases conversion rates. The treatment/control comparison is direct, and the model can control for confounding variables like user demographics and device type.
Common Mistakes & How to Avoid Them
Mistake 1: Using accuracy on imbalanced data
- Problem: If 99% of transactions are legitimate, predicting "not fraud" always gets 99% accuracy
- Solution: Use precision, recall, F1, or AUC-ROC. Consider resampling (SMOTE) or class weights.
Mistake 2: Not calibrating probabilities
- Problem: Logistic regression outputs may not be well-calibrated probabilities
- Solution: Use
CalibratedClassifierCVfor better probability estimates. Apply Platt scaling or isotonic regression.
Mistake 3: Ignoring multicollinearity
- Problem: Highly correlated features make coefficient interpretation unreliable
- Solution: Check VIF, remove redundant features, or use regularization.
Mistake 4: Using the default threshold (0.5)
- Problem: The optimal threshold depends on the cost of false positives vs false negatives
- Solution: Use precision-recall curves to find the optimal threshold for your specific use case.
Mistake 5: Not scaling features
- Problem: Regularization penalizes coefficient magnitude — features with larger scales get unfairly penalized
- Solution: Always standardize features before training logistic regression with regularization.
Interview Questions
Q1: Why is logistic regression called "regression" when it's a classification algorithm? A: It's a historical name. The algorithm models the probability of a binary outcome using a regression framework (linear function + sigmoid). The output is continuous (probability), but the decision is discrete (class label).
Q2: What is the relationship between logistic regression and Naive Bayes? A: Both are probabilistic classifiers. Logistic regression models directly (discriminative). Naive Bayes models and , then uses Bayes' theorem (generative). Logistic regression typically performs better because it doesn't assume feature independence.
Q3: How does the C parameter in sklearn's LogisticRegression work? A: C is the inverse of regularization strength: . Small C = strong regularization (simpler model). Large C = weak regularization (complex model). Default C=1.0 is a good starting point.
Q4: When would you use One-vs-Rest vs Softmax for multiclass? A: One-vs-Rest trains K binary classifiers — simpler, works with any binary classifier. Softmax (multinomial) trains one model with K outputs — more efficient, accounts for class competition. Softmax is preferred when classes are mutually exclusive.
Q5: How do you handle missing values in logistic regression? A: (1) Impute with mean/median/mode, (2) Use models that handle missing data natively (some tree-based models), (3) Create an indicator feature for missingness, (4) Use multiple imputation for better estimates.
Q6: What is the decision boundary of logistic regression? A: The decision boundary is the hyperplane — a linear boundary. On one side, (predict class 1). On the other, (predict class 0). For nonlinear boundaries, add polynomial features or use kernel methods.
Q7: How does logistic regression handle class imbalance?
A: Use the class_weight='balanced' parameter, which automatically adjusts weights inversely proportional to class frequencies. This penalizes misclassification of minority class more heavily. Alternatively, use SMOTE oversampling or threshold tuning.
Practice Exercise
Challenge: Implement Logistic Regression from Scratch
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, roc_auc_score
# Generate data
X, y = make_classification(n_samples=1000, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Standardize
X_train = (X_train - X_train.mean(axis=0)) / X_train.std(axis=0)
X_test = (X_test - X_test.mean(axis=0)) / X_test.std(axis=0)
# Sigmoid function
def sigmoid(z):
return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
# Cost function
def compute_cost(X, y, w, b):
m = len(y)
z = X @ w + b
h = sigmoid(z)
cost = -np.mean(y * np.log(h + 1e-15) + (1 - y) * np.log(1 - h + 1e-15))
return cost
# Gradient descent
def gradient_descent(X, y, lr=0.01, epochs=1000):
m, n = X.shape
w = np.zeros(n)
b = 0
for epoch in range(epochs):
z = X @ w + b
h = sigmoid(z)
# Gradients
dw = (1/m) * X.T @ (h - y)
db = (1/m) * np.sum(h - y)
# Update
w -= lr * dw
b -= lr * db
if epoch % 200 == 0:
cost = compute_cost(X, y, w, b)
print(f"Epoch {epoch:4d}: Cost = {cost:.4f}")
return w, b
# Train
w, b = gradient_descent(X_train, y_train, lr=0.1, epochs=1000)
# Predict
y_prob = sigmoid(X_test @ w + b)
y_pred = (y_prob >= 0.5).astype(int)
print(f"\nAccuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"AUC-ROC: {roc_auc_score(y_test, y_prob):.3f}")
# Examine coefficients
print("\nFeature importance (|w|):")
for i, (weight, importance) in enumerate(zip(w, np.abs(w))):
print(f" Feature {i}: w={weight:.4f}, |w|={importance:.4f}")
Bonus challenges:
- Implement L2 regularization by adding to the cost
- Experiment with different learning rates and plot convergence curves
- Implement multiclass logistic regression using One-vs-Rest
Comparison Table
Classification Algorithms Comparison
| Algorithm | Output | Decision Boundary | Speed | Interpretability |
|---|---|---|---|---|
| Logistic Regression | Probabilities | Linear | ★★★★★ | ★★★★★ |
| Naive Bayes | Probabilities (via Bayes) | Linear (or nonlinear) | ★★★★★ | ★★★★☆ |
| SVM (linear) | Class label (or distance) | Linear | ★★★★☆ | ★★★☆☆ |
| SVM (RBF) | Class label (or distance) | Nonlinear | ★★★☆☆ | ★★☆☆☆ |
| Decision Tree | Class label (or probability) | Axis-aligned | ★★★★☆ | ★★★★★ |
Key Formulas Reference
| Formula | Expression | Context |
|---|---|---|
| Sigmoid | Maps to (0,1) | |
| Prediction | Probability output | |
| Cross-Entropy | Cost function | |
| Gradient | Optimization | |
| Softmax | Multiclass |
Key Takeaways
What to Learn Next
-> Linear Regression From scatter plots to predictions — the simplest ML algorithm.
-> Naive Bayes Bayes' theorem in action — fast, simple, surprisingly powerful.
-> SVM Finding the optimal boundary — maximum margin classification.