Loss Functions: Design and Optimization

Module: Machine Learning | Difficulty: Advanced

Exponential Family Loss

Proper Scoring Rules

A scoring rule is proper if for all .

| Loss | Formula | Proper | Robust | |------|---------|--------|--------| | Squared | | Yes | No | | Log | | Yes | No | | Hinge | | No | Yes | | Huber | Mixed | No | Yes |

Huber Loss

Focal Loss

import numpy as np

class HuberLoss:
    def __init__(self, delta=1.0):
        self.delta = delta
    def loss(self, y, f):
        err = y - f
        is_small = np.abs(err) <= self.delta
        return np.where(is_small, 0.5*err**2, self.delta*np.abs(err) - 0.5*self.delta**2)
    def gradient(self, y, f):
        err = y - f
        is_small = np.abs(err) <= self.delta
        return np.where(is_small, -err, -self.delta*np.sign(err))

Research Insight: Focal loss was designed for class imbalance. The term down-weights easy examples, forcing the model to focus on hard examples. works best for most applications.