Variational Autoencoders — Deep Dive

VAEs are generative models that learn a smooth latent space by combining autoencoders with variational inference. Unlike GANs, they provide a principled probabilistic framework for generation.

From Autoencoder to VAE

DfAutoencoder

A standard autoencoder compresses input $x$ into a latent code $z$ (encoder) and reconstructs it (decoder). It learns a deterministic mapping but the latent space may have gaps, making generation difficult.

DfVariational Autoencoder (VAE)

A VAE (Kingma & Welling, 2014) learns a probabilistic latent space:

Encoder $q_\phi(z|x)$ : Maps input to a distribution (mean $\mu$ , variance $\sigma^2$ )
Decoder $p_\theta(x|z)$ : Reconstructs input from sampled latent
Prior $p(z) = \mathcal{N}(0, I)$ : Standard Gaussian

Instead of encoding to a point, VAE encodes to a distribution. Sampling from this distribution forces the latent space to be smooth and continuous.

Evidence Lower Bound (ELBO)

\mathcal{L}_{\text{VAE}} = \underbrace{-\mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]}_{\text{Reconstruction Loss}} + \underbrace{D_{KL}(q_\phi(z|x) \| p(z))}_{\text{KL Divergence}}

Reconstruction Loss

\mathcal{L}_{\text{recon}} = -\mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]

Here,

$q_\phi(z|x)$ =Encoder distribution (approximate posterior)
$p_\theta(x|z)$ =Decoder distribution (likelihood)
$\log p_\theta(x|z)$ =Log-likelihood of reconstruction

KL Divergence

D_{KL}(q_\phi(z|x) \| p(z)) = -\frac{1}{2} \sum_{j=1}^{J} \left(1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2\right)

Here,

$\mu_j$ =Mean of encoder output for dimension j
$\sigma_j^2$ =Variance of encoder output for dimension j
$J$ =Latent dimension

ℹ️ ELBO Derivation

\log p(x) = \mathbb{E}_{q_\phi(z|x)}\left[\log \frac{p_\theta(x,z)}{q_\phi(z|x)}\right] + D_{KL}(q_\phi(z|x) \| p_\theta(z|x))

Since KL divergence is always non-negative, the first term (ELBO) is a lower bound on $\log p(x)$ . Maximizing the ELBO simultaneously maximizes data likelihood and minimizes the gap to the true posterior.

Reparameterization Trick

ThReparameterization Trick

To backpropagate through a stochastic sampling operation, reparameterize the sampling as a deterministic transformation of noise:

z = \mu + \sigma \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)

This makes the sampling differentiable with respect to $\mu$ and $\sigma$ while maintaining the stochastic nature through $\epsilon$ .

Reparameterization

z = \mu + \sigma \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)

Here,

$\mu$ =Mean vector from encoder
$\sigma$ =Standard deviation from encoder
$\epsilon$ =Random noise (sampled during forward, not backpropagated)
$\odot$ =Element-wise multiplication

Variational Autoencoders — Deep Dive

Variational Autoencoders — Deep Dive

From Autoencoder to VAE

DfAutoencoder

DfVariational Autoencoder (VAE)

Evidence Lower Bound (ELBO)

Reconstruction Loss

KL Divergence

Reparameterization Trick

ThReparameterization Trick

Reparameterization

Need Expert Deep Learning Help?