DL Foundations
Loss Functions — The Compass That Guides Neural Network Training
Loss functions quantify how wrong a model's predictions are, defining the objective that optimization algorithms minimize. Choosing the right loss function is critical for effective training and determines what the model learns.
- Cross-Entropy Dominates — The standard loss for classification, equivalent to maximum likelihood estimation
- Focal Loss for Imbalance — Down-weights easy examples to focus on hard, rare cases in object detection
- Huber Loss for Robustness — Combines MSE and MAE to handle outliers in regression tasks
Loss Functions for Deep Learning — MSE, Cross-Entropy, Focal Loss and Beyond
Loss functions quantify how wrong a model's predictions are. Choosing the right loss function is critical for effective training.
Loss Function Taxonomy
Mean Squared Error (MSE)
Cross-Entropy Loss
Focal Loss
Huber Loss
Contrastive and Triplet Loss
Loss Function Comparison
Label Smoothing
Summary
- MSE for regression: quadratic penalty, sensitive to outliers
- Cross-entropy for classification: equivalent to maximum likelihood
- Focal loss handles class imbalance by down-weighting easy examples
- Huber loss combines MSE and MAE for robustness
- Triplet/contrastive loss for learning embeddings
- Choose the loss function based on your task, data distribution, and what you want to optimize