Energy-Based Models (EBMs)
1. The Energy Function Framework
Energy-based models assign unnormalized probabilities to configurations via an energy function :
where is the partition function (normalizing constant).
1.1 The Core Problem
The partition function is generally intractable:
This makes exact likelihood computation impossible, necessitating alternative training objectives.
1.2 Gibbs Distribution Perspective
The energy function defines a Gibbs measure:
The energy can be decomposed as:
where is the energy score (higher energy = lower probability).
2. The Score Function
2.1 Definition
The score function is the gradient of the log-density:
Since the partition function cancels:
2.2 Score Matching (Hyvärinen, 2005)
The objective minimizes the expected distance between model and data scores:
Expanding:
The denoising score matching (Vincent, 2011) estimator:
3. Energy-Based Model Landscape
<svg viewBox="0 0 800 400" xmlns="http://www.w3.org/2000/svg">
<rect width="800" height="400" fill="#f8fafc"/>
{/* Title */}
<text x="400" y="30" text-anchor="middle" fill="#f8fafc" font-family="monospace" font-size="16" font-weight="bold">Energy Landscape & Training</text>
{/* Energy Landscape */}
<g transform="translate(50, 50)">
<text x="150" y="0" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="12">Energy Landscape E(x)</text>
{/* Energy surface */}
<path d="M 0 250 Q 50 200 100 220 Q 150 150 200 180 Q 250 100 300 130 Q 350 160 400 120 Q 450 80 500 110 Q 550 140 600 100 Q 650 60 700 90"
fill="none" stroke="#3b82f6" stroke-width="3"/>
{/* Data points (low energy) */}
<circle cx="250" cy="105" r="6" fill="#22c55e"/>
<circle cx="260" cy="98" r="5" fill="#22c55e"/>
<circle cx="240" cy="108" r="5" fill="#22c55e"/>
<circle cx="500" cy="115" r="6" fill="#22c55e"/>
<circle cx="510" cy="108" r="5" fill="#22c55e"/>
<circle cx="490" cy="118" r="5" fill="#22c55e"/>
{/* High energy regions */}
<circle cx="100" cy="215" r="5" fill="#ec4899" opacity="0.7"/>
<circle cx="150" cy="155" r="5" fill="#ec4899" opacity="0.7"/>
<circle cx="350" cy="155" r="5" fill="#ec4899" opacity="0.7"/>
{/* Labels */}
<text x="250" y="135" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="9">Low E(x)</text>
<text x="100" y="240" text-anchor="middle" fill="#ec4899" font-family="monospace" font-size="9">High E(x)</text>
{/* Axes */}
<line x1="0" y1="270" x2="700" y2="270" stroke="#64748b" stroke-width="1"/>
<line x1="0" y1="50" x2="0" y2="270" stroke="#64748b" stroke-width="1"/>
<text x="350" y="290" text-anchor="middle" fill="#64748b" font-family="monospace" font-size="10">x</text>
<text x="-20" y="160" text-anchor="middle" fill="#64748b" font-family="monospace" font-size="10">E(x)</text>
</g>
{/* Contrastive Divergence */}
<g transform="translate(50, 320)">
<text x="350" y="0" text-anchor="middle" fill="#a855f7" font-family="monospace" font-size="12">Contrastive Divergence Training</text>
{/* Markov chain */}
<circle cx="50" cy="30" r="8" fill="#3b82f6"/>
<text x="50" y="34" text-anchor="middle" fill="white" font-family="monospace" font-size="8">x₀</text>
<line x1="60" y1="30" x2="90" y2="30" stroke="#64748b" stroke-width="1" marker-end="url(#2033_arrow)"/>
<text x="75" y="22" text-anchor="middle" fill="#f59e0b" font-family="monospace" font-size="8">p(h|x)</text>
<circle cx="110" cy="30" r="8" fill="#a855f7"/>
<text x="110" y="34" text-anchor="middle" fill="white" font-family="monospace" font-size="8">h₀</text>
<line x1="120" y1="30" x2="150" y2="30" stroke="#64748b" stroke-width="1" marker-end="url(#2033_arrow)"/>
<text x="135" y="22" text-anchor="middle" fill="#f59e0b" font-family="monospace" font-size="8">p(x|h)</text>
<circle cx="170" cy="30" r="8" fill="#3b82f6"/>
<text x="170" y="34" text-anchor="middle" fill="white" font-family="monospace" font-size="8">x₁</text>
<line x1="180" y1="30" x2="210" y2="30" stroke="#64748b" stroke-width="1" marker-end="url(#2033_arrow)"/>
<text x="195" y="22" text-anchor="middle" fill="#f59e0b" font-family="monospace" font-size="8">p(h|x)</text>
<circle cx="230" cy="30" r="8" fill="#a855f7"/>
<text x="230" y="34" text-anchor="middle" fill="white" font-family="monospace" font-size="8">h₁</text>
<line x1="240" y1="30" x2="270" y2="30" stroke="#64748b" stroke-width="1" marker-end="url(#2033_arrow)"/>
<text x="255" y="22" text-anchor="middle" fill="#f59e0b" font-family="monospace" font-size="8">p(x|h)</text>
<circle cx="290" cy="30" r="8" fill="#ec4899"/>
<text x="290" y="34" text-anchor="middle" fill="white" font-family="monospace" font-size="8">xₖ</text>
{/* Gradient arrows */}
<path d="M 170 45 L 50 45" stroke="#22c55e" stroke-width="2" marker-end="url(#2033_arrowGreen)"/>
<text x="110" y="58" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="8">-∇E(x₀) + ∇E(x₁)</text>
<path d="M 290 45 L 170 45" stroke="#22c55e" stroke-width="2" marker-end="url(#2033_arrowGreen)"/>
<text x="230" y="58" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="8">-∇E(x₁) + ∇E(xₖ)</text>
{/* Legend */}
<rect x="400" y="10" width="280" height="60" rx="5" fill="#f8fafc" stroke="#334155"/>
<circle cx="420" cy="30" r="6" fill="#3b82f6"/>
<text x="435" y="34" fill="#94a3b8" font-family="monospace" font-size="9">Data samples</text>
<circle cx="420" cy="50" r="6" fill="#a855f7"/>
<text x="435" y="54" fill="#94a3b8" font-family="monospace" font-size="9">Model samples</text>
<rect x="550" y="24" width="20" height="12" fill="#22c55e"/>
<text x="580" y="34" fill="#94a3b8" font-family="monospace" font-size="9">Gradient update</text>
</g>
<defs>
<marker id="2033_arrow" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
<polygon points="0 0, 10 3.5, 0 7" fill="#64748b"/>
</marker>
<marker id="2033_arrowGreen" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
<polygon points="0 0, 10 3.5, 0 7" fill="#22c55e"/>
</marker>
</defs>
</svg>
4. Contrastive Divergence
4.1 The CD-k Algorithm
Contrastive Divergence (Hinton, 2002) approximates the gradient of the log-likelihood:
The negative phase is approximated by running a Gibbs chain for steps starting from the data:
4.2 CD Objective
For :
4.3 Persistent CD (PCD)
In Persistent Contrastive Divergence (Tieleman, 2008), the Markov chain is maintained across training steps:
This provides a better approximation of the model distribution, especially for RBMs.
4.4 Stochastic Maximum Likelihood (SML)
PCD can be viewed as Stochastic Maximum Likelihood:
5. Score Matching Methods
5.1 Score Matching Objective
The optimal score function satisfies:
5.2 Sliced Score Matching
To avoid computing the full Hessian:
where is a random projection vector and is the Jacobian of the score.
5.3 Noise Contrastive Score Matching
Add noise to create a denoising problem:
6. Langevin Dynamics
6.1 Langevin Sampling
The Langevin dynamics sampler generates samples from :
where and is the step size.
As and , this converges to .
6.2 Annealed Langevin Dynamics
For better mixing, use a sequence of noise levels :
where decreases from to .
6.3 Underdamped Langevin Dynamics
For faster mixing in high dimensions:
where is the friction coefficient.
7. Training EBMs
7.1 Maximum Likelihood Training
7.2 Noise Contrastive Estimation (NCE)
Introduce a noise distribution and learn a discriminator:
where and is the sigmoid function.
7.3 InfoNCE / Contrastive Loss
8. JEPA Connection
8.1 Joint Embedding Predictive Architecture
JEPA (LeCun, 2022) uses energy-based formulation for self-supervised learning:
where are encoder networks mapping to a shared representation space.
8.2 The Energy Function
The energy between two views:
JEPA minimizes:
8.3 Predictive Coding Perspective
JEPA can be viewed as predictive coding in representation space:
where is a divergence measure (e.g., MSE, cosine distance).
8.4 VICReg Connection
VICReg (Bardes et al., 2022) extends JEPA with:
9. Advanced EBM Architectures
9.1 Sum-Product Networks
For tractable EBMs:
with sum nodes for mixture components and product nodes for independence.
9.2 Deep EBMs (Grathwohl et al., 2019)
Use steering to improve EBM training:
9.3 Flow-Energy Models
Combine flows with EBMs:
where is a flow transformation and is a simple energy model.
10. Connections to Other Models
10.1 EBMs vs. GANs
| Aspect | EBM | GAN |
|---|---|---|
| Training | Score matching, CD | Minimax game |
| Sampling | MCMC | Generator forward pass |
| Density | Approximate | Not available |
10.2 EBMs vs. VAEs
| Aspect | EBM | VAE |
|---|---|---|
| Density | Unnormalized | Variational lower bound |
| Latent | Optional | Required |
| Flexibility | High | Moderate |
10.3 EBMs vs. Diffusion Models
| Aspect | EBM | Diffusion |
|---|---|---|
| Score | Directly modeled | Learned via denoising |
| Sampling | Langevin dynamics | Reverse diffusion |
| Training | Score matching | Denoising objective |
Energy-based models provide a unifying framework for understanding the relationship between different generative modeling approaches, with the score function as the central quantity connecting EBMs to diffusion models, contrastive learning, and self-supervised methods.