DL Foundations
Loss Functions - The Compass That Guides Neural Network Training
Loss functions quantify how wrong a model's predictions are, defining the objective that optimization algorithms minimize. Choosing the right loss function is critical for effective training and determines what the model learns.
- Cross-Entropy Dominates - The standard loss for classification, equivalent to maximum likelihood estimation
- Focal Loss for Imbalance - Down-weights easy examples to focus on hard, rare cases in object detection
- Huber Loss for Robustness - Combines MSE and MAE to handle outliers in regression tasks
Loss Functions for Deep Learning - MSE, Cross-Entropy, Focal Loss and Beyond
What You'll Learn
- 1Derive MSE and BCE as negative log-likelihoods of Gaussian and Bernoulli models
- 2Compute the gradient of each loss with respect to logits and predictions
- 3Tune focal loss gamma on a 100 to 1 imbalanced detection set
- 4Pick a loss from the task, the label structure, and the presence of outliers
Loss functions quantify how wrong a model's predictions are. Choosing the right loss function is critical for effective training.
Loss Function Taxonomy
Mean Squared Error (MSE)
MSE is not an arbitrary choice; it is the negative log-likelihood of a model whose errors are Gaussian with fixed variance :
Drop the constant, set to , and the sum of squares falls out. The gradient is linear in the residual, which makes step size proportional to the error:
A residual of 10 contributes 20 times the gradient of a residual of 1, so a single mislabeled outlier can dominate an epoch. That sensitivity is exactly what Huber and MAE exist to blunt.
Cross-Entropy Loss
Feeding softmax probabilities in gives the numerically safe form used in code, where only the true logit and one log-sum-exp survive:
Binary cross-entropy with a single sigmoid logit simplifies the same way:
The first expression is the softplus-based implementation that does not overflow for large negative . Note the gradient is again prediction minus target: the pairing of sigmoid with BCE and softmax with CE is what makes classification training a single clean subtraction per example.
Focal Loss
Huber Loss
Because the slope saturates at , one bad label cannot steer the epoch:
With an error of 100 contributes the same gradient magnitude as an error of 1, while errors below 1 keep the smooth quadratic behavior that optimizers like.
Contrastive and Triplet Loss
Loss Function Comparison
Label Smoothing
Expanding the cross-entropy with smoothed targets shows the extra term clearly:
The second term keeps every class logit alive, so the model can never drive to 1 and starve the remaining weights of gradient. With the target logit receives 90 percent of the mass and the rest is shared, which calibrates the final probabilities and typically buys a few tenths of a percent of top-1 accuracy on ImageNet sized tasks.
Summary
- MSE for regression: quadratic penalty, sensitive to outliers
- Cross-entropy for classification: equivalent to maximum likelihood
- Focal loss handles class imbalance by down-weighting easy examples
- Huber loss combines MSE and MAE for robustness
- Triplet/contrastive loss for learning embeddings
- Choose the loss function based on your task, data distribution, and what you want to optimize
Next: Optimizers for Deep Learning
Common Mistakes
Using MSE on a classification target
Fix: Squared error on probabilities gives gradients that vanish as predictions saturate; use cross-entropy, whose gradient stays proportional to p minus y.
Applying softmax and then BCE per class
Fix: Softmax outputs sum to 1 so the classes are not independent; either softmax with CE or independent logits with BCE, never a mix.
Tuning alpha but forgetting gamma in focal loss
Fix: gamma is the re-weighting knob and alpha only rescales classes; sweep gamma over 1, 2, 5 first, then set alpha to the inverse class frequency.
Best Practices
- Read your loss as a negative log-likelihood and confirm the noise model matches the data
- Compute losses from logits with the fused framework kernel so log-sum-exp stays stable
- Log the loss distribution, not just its mean, so a handful of outliers are visible
Key Takeaways
- MSE is Gaussian negative log-likelihood with gradient 2(y_hat - y)/N, and it overweights outliers
- Softmax plus CE backpropagates to p minus y; BCE with sigmoid gives the same one-line gradient
- Focal loss multiplies CE by (1 - p_t)^gamma so a 0.9 confident example keeps only 1 percent of its weight at gamma = 2
- Huber caps the gradient at delta, and label smoothing stops targets from pushing logits to infinity
Frequently Asked Questions
Why is cross-entropy preferred over MSE for classification?
Two reasons. Probabilities live on a bounded simplex, so MSE gradients shrink as predictions saturate and learning stalls; cross-entropy keeps a gradient of that stays large exactly when the prediction is wrong. And cross-entropy is the negative log-likelihood of a categorical model, so minimizing it is maximum likelihood estimation rather than a heuristic score.
What does focal loss actually change?
It multiplies each example's cross-entropy by , a per-example weight that depends on confidence. Easy examples ( near 1) collapse toward zero contribution while hard ones survive, so the effective class balance shifts toward rare positives without oversampling or class weights. It costs nothing at training time beyond one extra power operation.
When does Huber beat MSE in practice?
Whenever the target has heavy tails or the labels contain mistakes: house prices, sensor spikes, bounding box coordinates. MSE grows as the square of the residual, so a single 10x outlier can outweigh a thousand clean examples; Huber grows only linearly past , capping that example's gradient while keeping the smooth quadratic regime where most of the data lives.
Interview Questions
Q1: Show that BCE is the negative log-likelihood of a Bernoulli model.
For y in {0,1} with predicted probability p, the Bernoulli likelihood is p^y (1-p)^(1-y). Taking the negative log gives -y log p - (1-y) log(1-p), and averaging over N samples is exactly the BCE formula, so cross-entropy training is maximum likelihood under that noise model.
Q2: Derive the gradient of softmax plus cross-entropy with respect to the logits.
L = -z_t + log sum_j exp z_j. Differentiating gives softmax(z)_i minus 1 for i = t and softmax(z)_i otherwise, which is p_i minus y_i in both cases. That one subtraction is why the pair is fused in every framework.
Q3: You have 50,000 background and 500 object crops. What loss and settings?
Focal loss with gamma = 2 and alpha = 0.25, as in RetinaNet. Plain CE at 100 to 1 lets background dominate; focal down-weights the confident background crops by roughly (1 - 0.9)^2 = 0.01 while keeping the rare hard objects near full weight.