Numerical Integration
Definitions
Formulas
Trapezoidal Rule
Derivation
From the single-interval trapezoidal rule applied to each subinterval and summed:
- Total = Σᵢ₌₀ⁿ⁻¹ h·(f(xᵢ) + f(xᵢ₊₁))/2
- Interior points appear twice (once as right endpoint, once as left)
- Endpoints appear once
- Result: h/2·[f(x₀) + 2f(x₁) + ... + 2f(x₋₁) + f(xₙ)]
Simpson's Rule
Weight Pattern
For n = 6 (7 nodes): weights are [1, 4, 2, 4, 2, 4, 1] × h/3 Pattern: 1, 4, 2, 4, 2, 4, ..., 4, 1
Gaussian Quadrature
Legendre Polynomial Nodes and Weights
| n | Nodes (xᵢ) | Weights (wᵢ) | Exact for degree |
|---|---|---|---|
| 1 | 0 | 2 | 1 |
| 2 | ±1/√3 | 1 | 3 |
| 3 | 0, ±√(3/5) | 8/9, 5/9 | 5 |
| 4 | ±0.339981, ±0.861136 | 0.652145, 0.347855 | 7 |
Transformation
To integrate over [a, b] instead of [−1, 1]: x = ((b−a)t + (b+a))/2, dx = (b−a)/2 dt
Examples
Theorems
Python Implementation
import numpy as np
from typing import Callable, Tuple
class NumericalIntegrator:
"""Comprehensive numerical integration toolkit."""
@staticmethod
def trapezoidal(f: Callable, a: float, b: float,
n: int = 1000) -> Tuple[float, float]:
"""Composite trapezoidal rule. Returns (result, error_estimate)."""
x = np.linspace(a, b, n + 1)
y = f(x)
h = (b - a) / n
result = h * (y[0] / 2 + np.sum(y[1:-1]) + y[-1] / 2)
# Error estimate via Richardson extrapolation
x2 = np.linspace(a, b, 2 * n + 1)
y2 = f(x2)
h2 = h / 2
result2 = h2 * (y2[0] / 2 + np.sum(y2[1:-1]) + y2[-1] / 2)
error = abs(result2 - result) / 3 # Richardson estimate
return result, error
@staticmethod
def simpson(f: Callable, a: float, b: float,
n: int = 1000) -> Tuple[float, float]:
"""Composite Simpson's rule (n forced even). Returns (result, error)."""
if n % 2 != 0:
n += 1
x = np.linspace(a, b, n + 1)
y = f(x)
h = (b - a) / n
result = h / 3 * (y[0] + 4 * np.sum(y[1::2]) +
2 * np.sum(y[2:-1:2]) + y[-1])
# Error estimate
n2 = 2 * n
x2 = np.linspace(a, b, n2 + 1)
y2 = f(x2)
h2 = (b - a) / n2
result2 = h2 / 3 * (y2[0] + 4 * np.sum(y2[1::2]) +
2 * np.sum(y2[2:-1:2]) + y2[-1])
error = abs(result2 - result) / 15 # Richardson estimate
return result, error
@staticmethod
def gauss_legendre(f: Callable, a: float, b: float,
n: int = 5) -> float:
"""n-point Gauss-Legendre quadrature."""
nodes, weights = np.polynomial.legendre.leggauss(n)
t = 0.5 * (b - a) * nodes + 0.5 * (b + a)
w = 0.5 * (b - a) * weights
return np.sum(w * f(t))
@staticmethod
def romberg(f: Callable, a: float, b: float,
max_level: int = 10) -> np.ndarray:
"""Romberg integration using Richardson extrapolation."""
R = np.zeros((max_level + 1, max_level + 1))
# Trapezoidal estimates at various levels
n = 1
h = b - a
R[0, 0] = h / 2 * (f(a) + f(b))
for i in range(1, max_level + 1):
n = 2 ** i
h = (b - a) / n
# Add new midpoints
x_new = np.arange(1, n, 2) * h + a
R[i, 0] = 0.5 * R[i-1, 0] + h * np.sum(f(x_new))
# Richardson extrapolation
for j in range(1, i + 1):
R[i, j] = R[i, j-1] + (R[i, j-1] - R[i-1, j-1]) / (4**j - 1)
return R
@staticmethod
def monte_carlo(f: Callable, domain: list,
n_samples: int = 100000) -> Tuple[float, float]:
"""Monte Carlo integration over a rectangular domain."""
a, b = domain[0], domain[1]
volume = b - a
x = np.random.uniform(a, b, n_samples)
values = f(x)
result = volume * np.mean(values)
std_err = volume * np.std(values) / np.sqrt(n_samples)
return result, std_err
def demonstrate_integration():
"""Full demonstration of all integration methods."""
print("=" * 60)
print("NUMERICAL INTEGRATION METHOD COMPARISON")
print("=" * 60)
# Test 1: Smooth function
f1 = lambda x: np.exp(-x**2)
exact1 = np.sqrt(np.pi) # ∫₋∞^∞ e^(-x²) dx = √π
# But we integrate from -5 to 5 (close enough)
from scipy.integrate import quad
exact1_quad, _ = quad(f1, -5, 5)
print(f"\n∫₋₅⁵ e^(-x²) dx")
print(f"scipy quad: {exact1_quad:.12f}")
integrator = NumericalIntegrator()
for n in [10, 50, 200]:
trap, err = integrator.trapezoidal(f1, -5, 5, n)
simp, err = integrator.simpson(f1, -5, 5, n)
print(f" n={n:>3}: Trapezoidal {trap:.10f} (err {err:.2e}) | "
f"Simpson {simp:.10f} (err {err:.2e})")
# Gauss-Legendre
for n in [3, 5, 10]:
gl = integrator.gauss_legendre(f1, -5, 5, n)
print(f" Gauss-Legendre (n={n}): {gl:.10f}")
# Romberg
R = integrator.romberg(f1, -5, 5, max_level=6)
print(f"\nRomberg integration:")
for i in range(min(7, R.shape[0])):
print(f" Level {i}: {R[i, i]:.12f}")
# Test 2: Oscillatory function
print(f"\n{'=' * 60}")
f2 = lambda x: np.sin(100 * x) / (1 + x**2)
mc_result, mc_err = integrator.monte_carlo(f2, [-10, 10], 1_000_000)
quad_result, _ = quad(f2, -10, 10)
print(f"∫₋₁₀¹⁰ sin(100x)/(1+x²) dx")
print(f"scipy quad: {quad_result:.10f}")
print(f"Monte Carlo: {mc_result:.10f} (±{mc_err:.2e})")
demonstrate_integration()
Applications in AI/ML
Variational Inference
In Bayesian neural networks, the posterior P(θ|D) is intractable. Variational inference approximates it with a tractable distribution q(θ) and minimizes KL(q||p), which requires computing integrals like E_q[log p(D|θ)] using quadrature or Monte Carlo.
Monte Carlo in ML
| Application | Method | Description |
|---|---|---|
| Bayesian inference | MCMC, VI | Sample from posterior distributions |
| Policy gradients | REINFORCE | Monte Carlo estimation of expected returns |
| Variational autoencoders | Reparameterization | Monte Carlo gradient estimation |
| Normalizing flows | Importance sampling | Monte Carlo estimation of likelihood |
| GANs | Monte Carlo | Estimate partition function |
Loss Function Integration
Some loss functions require integration: Expected Calibration Error (ECE) integrates over confidence bins, and proper scoring rules integrate over predicted distributions.
Common Mistakes
| Mistake | Problem | Solution |
|---|---|---|
| Odd n for Simpson's rule | Formula requires even number of intervals | Round n up to nearest even number |
| Ignoring error estimate | Trusting result without verification | Always compute error estimate or compare with finer grid |
| Uniform spacing for peaked functions | Wasted evaluations in flat regions | Use adaptive quadrature |
| Low N for Monte Carlo | Slow convergence (O(1/√N)) | Use variance reduction or quasi-Monte Carlo |
| Integrating over singularities | Infinite error | Transform variable or split integral |
| Not checking integrability | Divergent integral returns wrong value | Verify convergence before trusting result |
| Using Simpson's for oscillatory | O(h⁴) not enough for highly oscillatory | Use specialized oscillatory quadrature |
| Ignoring roundoff in long sums | Kahan summation may be needed | Use compensated summation for many terms |
Interview Questions
Q1: Why is Gaussian quadrature more accurate than equally-spaced rules? A: Gaussian quadrature optimizes both node placement and weights to maximize the degree of exactness. With n nodes it integrates polynomials of degree 2n−1 exactly, while equally-spaced rules (Newton-Cotes) with n nodes achieve at most degree n (or n+1 for Simpson's). The trade-off is that Gaussian nodes are irrational, requiring more complex implementation.
Q2: When would you use Monte Carlo over deterministic quadrature? A: Monte Carlo is preferred for high-dimensional integrals (d > 4), where deterministic methods suffer the curse of dimensionality. It is also useful when the integrand is noisy, discontinuous, or only available through simulations. For low-dimensional smooth functions, deterministic methods are far more efficient.
Q3: How does adaptive quadrature decide where to subdivide? A: Adaptive quadrature estimates the local error on each subinterval. If the error exceeds the tolerance, the interval is split (usually in half). The process recurses until all subintervals meet the error tolerance. This concentrates evaluations where the integrand varies rapidly.
Q4: Explain the relationship between numerical integration and numerical differentiation. A: Numerical differentiation amplifies noise (it's an ill-conditioned problem), while numerical integration smooths it (it's well-conditioned). The error in numerical differentiation is O(ε/h) from roundoff plus O(h^p) from truncation, creating an optimal h ≈ √ε. Integration error simply decreases with h.
Q5: What is Richardson extrapolation and how is it used in Romberg integration? A: Richardson extrapolation combines numerical results at different step sizes to eliminate leading error terms. If R(h) = I + c₁h^p + ..., then combining R(h) and R(h/2) eliminates the h^p term, achieving higher accuracy. Romberg integration applies this recursively to trapezoidal rule estimates, achieving exponential convergence for smooth functions.
Q6: How do you handle integrable singularities? A: Strategies include: (1) variable substitution to remove the singularity (e.g., t = √x for x^(-1/2) singularity), (2) splitting the integral and handling the singular part analytically, (3) using Gauss-Jacobi or other specialized quadrature designed for singular weight functions, (4) adaptive quadrature which may concentrate points near the singularity.
Practice Problems
Quick Reference
Cross-References
- 086-Floating-Point — Numerical integration error is bounded by floating point precision; long summations need Kahan summation
- 087-Root-Finding — Adaptive quadrature uses error estimation similar to root-finding convergence criteria
- 089-Interpolation — Quadrature rules are derived by integrating interpolating polynomials; Gaussian quadrature uses optimal node placement
- 090-Error Analysis — Truncation error analysis determines the order of convergence for integration methods
- Monte Carlo Methods — Integration is the foundation of Monte Carlo simulation in physics, finance, and ML
- ODE Solvers: Every ODE solver (Euler, Runge-Kutta) uses quadrature internally to advance the solution