Logistic Regression: Sigmoid, Decision Boundary and Multi-class
The Bridge from Regression to Classification
Logistic regression extends linear regression to classification problems. Despite its name, it's a classification algorithm that predicts probabilities using the sigmoid function.
From Linear to Logistic: The Transformation
How this diagram works: This side-by-side comparison illustrates the critical transformation from linear regression to logistic regression. On the left, linear regression predicts unbounded values (like ŷ = 1.8) that go outside the valid probability range making it unsuitable for classification. On the right, the sigmoid function f(z) squashes any input into the [0, 1] range, producing valid probabilities. The S-shaped curve creates a natural decision boundary at 0.5, where points above the threshold are classified as Class 1 and below as Class 0.
1. The Sigmoid Function
Mathematical Definition
Sigmoid (Logistic) Function:
Properties:
- Output range: always a valid probability
- Symmetry:
- Derivative:
- At z = 0: (decision boundary)
- Asymptotes: ,
Derivative of Sigmoid
The derivative has an elegant form that enables efficient backpropagation:
Derivation:
Key Insight: The derivative is expressed in terms of the function itself no recomputation needed!
Log-Odds Interpretation
From Probability to Log-Odds:
Interpretation:
- = odds (ratio of success to failure)
- = log-odds (logit function)
- = change in log-odds for unit increase in
- = odds ratio multiplicative effect on odds
| Probability | Odds | Log-Odds |
|---|---|---|
| 0.01 | 0.0101 | ≈4.60 |
| 0.1 | 0.111 | ≈2.20 |
| 0.3 | 0.429 | ≈0.85 |
| 0.5 | 1.0 | 0.0 |
| 0.7 | 2.333 | 0.85 |
| 0.9 | 9.0 | 2.20 |
| 0.99 | 99.0 | 4.60 |
2. Decision Boundary
Linear Decision Boundary
The decision boundary is the hypersurface where , which occurs when .
Binary Classification (2D Features):
This is always a line (hyperplane in higher dimensions).
Non-Linear Decision Boundaries
By adding polynomial features, logistic regression can learn non-linear boundaries:
Polynomial Features:
This creates elliptical, parabolic, or other conic section boundaries.
3. Cost Function: Binary Cross-Entropy
Why Not MSE for Classification?
MSE with Sigmoid Creates Non-Convex Loss:
- Problem: Multiple local minima make optimization unreliable
- Gradient: Near saturation ( or ), gradient → slow learning
Binary Cross-Entropy Derivation
Single Sample Loss:
For : → penalizes low confidence predictions
For : → penalizes high confidence wrong predictions
Combined:
Average Cross-Entropy Loss
Full Cost Function:
Where
Gradient (same form as linear regression!):
4. Maximum Likelihood Estimation
Likelihood Function
Bernoulli Likelihood:
Each observation follows
Joint Likelihood (i.i.d. samples):
Log-Likelihood (easier to optimize):
Negative Log-L likelihood = Cross-Entropy Loss:
Gradient Derivation
Chain Rule Application:
Where:
Result:
The terms cancel beautifully!
5. Multi-class Classification
One-vs-Rest (OvR)
Train binary classifiers, one per class:
Softmax (Multinomial Logistic Regression)
For direct multi-class modeling, use the softmax function:
Softmax Function:
Properties:
- (valid probability distribution)
- Reduces to sigmoid when
- Monotonic: higher logit → higher probability
- Differentiable everywhere
Cross-Entropy for Multi-class
Categorical Cross-Entropy Loss:
Where is 1 if sample belongs to class , else 0 (one-hot encoded).
Gradient for class k:
6. Regularization
L1 Regularization (Lasso)
L1 Penalty:
Effect: Drives some coefficients to exactly zero → feature selection
** Interpretation:** Diamond constraint region → corner solutions
L2 Regularization (Ridge)
L2 Penalty:
Effect: Shrinks all coefficients toward zero, none exactly zero
Geometric Interpretation: Circular constraint region → smooth solutions
Elastic Net
Combined Penalty:
Advantage: Handles correlated features better than L1 alone
7. Complete Python Implementation
Binary Classification with Evaluation
Feature Importance and Odds Ratios
Decision Boundary Visualization
ROC Curve and AUC
8. Threshold Optimization
Default threshold = 0.5 is not always optimal!
Cost-Sensitive Threshold Selection:
Where = cost of false negative (missed disease), = cost of false alarm.
Key Takeaways
- Sigmoid function maps any real number to enables probability interpretation
- Cross-entropy loss is convex → guaranteed global minimum with gradient descent
- Decision boundary is always linear in feature space (use polynomial features for non-linear)
- Coefficient interpretation: = odds ratio for one-unit increase in
- Multi-class: Use softmax for direct multiclass, or OvR for binary decomposition
- Regularization: L1 for feature selection, L2 for smooth shrinkage, Elastic Net for both
- Threshold tuning is critical default 0.5 may not be optimal for imbalanced data
Practice Exercises
Exercise 1: Binary Classification Pipeline
Exercise 2: Multi-class Problem
Exercise 3: Cost-Sensitive Learning
- Train a model with
class_weight='balanced' - Compare confusion matrices with unweighted model
- In which scenarios is the weighted model better?
Exercise 4: Polynomial Features
Discussion Questions
- When would you prioritize recall over precision (and vice versa)?
- Why might AUC be preferred over accuracy for imbalanced datasets?
- How does regularization affect logistic regression coefficients and interpretability?
- Under what assumptions does logistic regression fail?