Interview Question

"Explain the decision boundary in logistic regression. Why do we use the logistic (sigmoid) function instead of a linear function for classification? How do you extend logistic regression to multiclass problems?"

Difficulty: Medium | Frequently asked at Meta, Microsoft, Amazon

Theoretical Foundation

The Logistic Model

Logistic regression models the probability that a binary outcome $y \in \{0, 1\}$ occurs given input features $x$:

where $z = \beta^T x + \beta_0$ is the linear combination of features and the sigmoid function $\sigma(z)$ maps any real number to the interval $(0, 1)$.

Properties of the sigmoid function:

• $\sigma(0) = 0.5$ (decision boundary)
• $\sigma(-z) = 1 - \sigma(z)$ (symmetry)
• $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ (efficient gradient computation)
• As $z \to \infty$, $\sigma(z) \to 1$
• As $z \to -\infty$, $\sigma(z) \to 0$

Why Not Use Linear Regression for Classification?

There are three fundamental problems:

1. Unbounded Output: Linear regression predicts values in $(-\infty, \infty)$, but probabilities must be in $[0, 1]$

2. Non-normal Errors: The error term $\epsilon$ is not normally distributed for binary outcomes (it follows a Bernoulli distribution)

3. Heteroscedasticity: The variance of the error term depends on the input, violating OLS assumptions

The Decision Boundary

The decision boundary is the hypersurface where the model switches between predicting class 0 and class 1. For logistic regression:

This is because:

• When $\beta^T x + \beta_0 > 0$: $P(y=1|x) > 0.5$ → Predict class 1
• When $\beta^T x + \beta_0 < 0$: $P(y=1|x) < 0.5$ → Predict class 0
• When $\beta^T x + \beta_0 = 0$: $P(y=1|x) = 0.5$ → Boundary

The decision boundary is always linear (a hyperplane in feature space). This is both a strength (simple, interpretable) and a limitation (cannot capture non-linear relationships without feature engineering).

The Cost Function

Logistic regression uses Binary Cross-Entropy (BCE) loss, also called log loss:

where $\hat{p}_i = \sigma(\beta^T x_i + \beta_0)$.

Why not use Mean Squared Error?

If we used MSE: $J(\beta) = \frac{1}{n} \sum_{i=1}^{n} (y_i - \sigma(\beta^T x_i))^2$

The problem is that MSE creates a non-convex loss surface with local minima, making optimization difficult. BCE is convex, guaranteeing a global minimum.

Mathematical Derivation of BCE:

From maximum likelihood estimation:

Taking the log:

Minimizing the negative log-likelihood gives us BCE.

Gradient of the Cost Function

The gradient has a beautiful form:

This is identical in form to the OLS gradient, which is why logistic regression can be solved with similar optimization algorithms.

Optimization Algorithms

1. Gradient Descent:

2. Newton-Raphson (Second-Order):

where $H$ is the Hessian matrix. Faster convergence but $O(p^2)$ per iteration.

3. L-BFGS (Limited-memory BFGS):

• Approximates the Hessian using limited memory
• Default in scikit-learn
• Good balance of speed and memory efficiency

4. Coordinate Descent (for regularized logistic regression):

• Used when L1/L2 penalties are added
• Solves one coordinate at a time
• Very efficient for high-dimensional sparse data

Multiclass Extensions

One-vs-Rest (OvR) / One-vs-All

Trains $K$ binary classifiers, one for each class:

Prediction: Choose the class with the highest probability.

Pros: Simple, parallelizable Cons: Class probabilities don't sum to 1; can be suboptimal when classes are not balanced

Multinomial Logistic Regression (Softmax)

Directly models $K$ classes using the softmax function:

Properties:

• Probabilities sum to 1: $\sum_{k=1}^{K} P(y=k|x) = 1$
• Requires $K$ weight vectors (or $K-1$ if one class is the reference)
• Uses categorical cross-entropy loss:

Code Implementation

Explanation of Code

1. Decision Boundary: Visualizes the linear decision boundary and shows the equation of the separating hyperplane.

2. Probability Output: Demonstrates how logistic regression outputs calibrated probabilities and how different thresholds affect predictions.

3. Multiclass Comparison: Compares One-vs-Rest vs Multinomial (Softmax) approaches, showing Softmax produces properly calibrated probabilities.

4. Cost Function: Illustrates why BCE is preferred over MSE for classification, showing BCE penalizes confident wrong predictions more heavily.

5. Regularization: Shows how C (inverse of λ) controls model complexity in logistic regression.

Real-World Applications

Meta: Content Ranking

Meta uses logistic regression (and its neural network extensions) for:

• News Feed Ranking: Predicting probability of user engagement
• Ad Targeting: Estimating click-through rates (CTR)
• Content Moderation: Classifying potentially harmful content

Microsoft: Spam Detection

Microsoft's email spam filters use multinomial logistic regression with:

• TF-IDF features from email text
• Header features (sender reputation, time sent)
• Behavioral features (sender-recipient interaction history)

Industry Best Practices

1. Feature Scaling: Always standardize features before logistic regression
2. Class Imbalance: Use class_weight='balanced' or SMOTE
3. Multicollinearity: Check VIF values; use regularization if VIF > 10
4. Model Calibration: Use CalibratedClassifierCV if probabilities need to be well-calibrated

Common Follow-Up Questions

Q1: Why is the sigmoid function preferred over other activation functions for binary classification?

The sigmoid function is preferred because:

• It's the inverse of the logit function, giving a natural probabilistic interpretation
• Its derivative $\sigma'(z) = \sigma(z)(1-\sigma(z))$ is easy to compute
• It arises naturally from the exponential family distribution
• It ensures outputs are in $[0, 1]$

Q2: How do you handle multiclass problems where classes are not mutually exclusive?

Use One-vs-Rest (OvR) instead of Softmax. Softmax assumes mutual exclusivity (probabilities sum to 1). For multi-label classification, train independent binary classifiers for each class.

Q3: What is the connection between logistic regression and neural networks?

Logistic regression is equivalent to a single-layer neural network with:

• One output neuron
• Sigmoid activation function
• Binary cross-entropy loss

Deep learning extends this by adding hidden layers and non-linear activations.

Q4: How do you detect and handle class imbalance in logistic regression?

1. Resampling: SMOTE (oversampling minority) or undersampling majority
2. Class weights: Set class_weight='balanced' in sklearn
3. Threshold tuning: Lower threshold to increase recall
4. Metrics: Use F1, AUC-PR instead of accuracy

Company-Specific Tips

Meta Interview Tips

• Discuss online learning variants for streaming data
• Be ready to explain probability calibration techniques
• Mention A/B testing frameworks for model comparison
• Talk about feature importance in high-dimensional sparse data

Microsoft Interview Tips

• Focus on interpretability requirements in regulated industries
• Discuss model serving at scale (batch vs real-time)
• Be prepared to explain regularization choices in production
• Mention monitoring for model drift in live systems

Related Topics

• Linear Regression Interview Guide
• Evaluation Metrics Deep Dive
• Feature Engineering Best Practices
• Neural Networks Introduction