KL Divergence

KL Divergence

Historical Context

Core Definitions

Key Formulas

Properties and Theorems

Worked Examples

Python Implementation

import numpy as np
from scipy import stats

def kl_divergence_discrete(p, q):
    """Compute KL divergence D_KL(P || Q) for discrete distributions."""
    p, q = np.array(p, dtype=float), np.array(q, dtype=float)
    # Filter where p > 0 to avoid log(0)
    mask = p > 0
    p, q = p[mask], q[mask]
    # Avoid division by zero
    q = np.where(q > 0, q, 1e-10)
    return np.sum(p * np.log(p / q))

def kl_divergence_gaussian(mu0, sigma0, mu1, sigma1):
    """Compute KL divergence between two univariate Gaussians."""
    return (np.log(sigma1 / sigma0) +
            (sigma0**2 + (mu0 - mu1)**2) / (2 * sigma1**2) - 0.5)

def reverse_kl_gaussian(mu0, sigma0, mu1, sigma1):
    """Compute D_KL(N(mu1, sigma1^2) || N(mu0, sigma0^2))."""
    return kl_divergence_gaussian(mu1, sigma1, mu0, sigma0)

def vae_kl_term(mu, log_var):
    """Compute KL divergence for VAE: D_KL(q(z|x) || N(0,I))."""
    # q(z|x) = N(mu, exp(log_var))
    # p(z) = N(0, 1)
    return -0.5 * np.sum(1 + log_var - mu**2 - np.exp(log_var))

# --- Examples ---
# Discrete distributions
p = [0.9, 0.1]
q = [0.5, 0.5]
print(f"KL(P||Q): {kl_divergence_discrete(p, q):.4f}")
print(f"KL(Q||P): {kl_divergence_discrete(q, p):.4f}")

# Gaussian KL
print(f"KL(N(0,1)||N(0,2)): {kl_divergence_gaussian(0, 1, 0, 2):.4f}")
print(f"KL(N(0,2)||N(0,1)): {kl_divergence_gaussian(0, 2, 0, 1):.4f}")

# VAE KL term
mu = np.array([0.5, -0.3])
log_var = np.array([-0.5, -1.0])
print(f"VAE KL term: {vae_kl_term(mu, log_var):.4f}")

Applications in AI/ML

Common Mistakes

Interview Questions

Q1: Why is KL divergence asymmetric? A: averages over , while averages over . These weight different regions of the space differently, leading to different values.

Q2: What's the difference between forward and reverse KL? A: Forward KL is mean-seeking (spreads to cover all of ). Reverse KL is mode-seeking (concentrates on one mode of ). VAEs use forward KL for regularization.

Q3: Can KL divergence be infinite? A: Yes. If has support where doesn't ( but ), the KL divergence is infinite. This is why it's important to ensure has support at least as wide as .

Q4: Why not just use Euclidean distance between distributions? A: Euclidean distance ignores the probability structure. Two distributions can be close in Euclidean distance but have very different shapes. KL divergence accounts for the actual probability values.

Q5: How is KL divergence used in VAEs? A: The VAE loss includes where . This regularizes the latent space, ensuring the encoder produces distributions close to the prior, enabling smooth interpolation and generation.

Practice Problems

Variants and Related Divergences

Quick Reference

Cross-References

• 081 - Entropy — — KL is the difference between cross-entropy and entropy.
• 082 - Mutual Information — — MI is a special case of KL divergence.
• 084 - Cross-Entropy — Cross-entropy loss = entropy + KL divergence. Minimizing CE is equivalent to minimizing KL when entropy is fixed.
• 085 - Applications — VAEs use KL regularization, EM algorithm uses KL in E-step, distillation uses KL to match teacher.

Common Pitfalls in Implementation

Summary