🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
Search courses…
💼 Servicesℹ️ About✉️ ContactView Pricing Plansfrom $10

Numerical Integration

Numerical MethodsIntegration🟢 Free Lesson

Advertisement

Numerical Integration


Definitions


Formulas


Trapezoidal Rule

Derivation

From the single-interval trapezoidal rule applied to each subinterval and summed:

  • Total = Σᵢ₌₀ⁿ⁻¹ h·(f(xᵢ) + f(xᵢ₊₁))/2
  • Interior points appear twice (once as right endpoint, once as left)
  • Endpoints appear once
  • Result: h/2·[f(x₀) + 2f(x₁) + ... + 2f(x₋₁) + f(xₙ)]

Simpson's Rule

Weight Pattern

For n = 6 (7 nodes): weights are [1, 4, 2, 4, 2, 4, 1] × h/3 Pattern: 1, 4, 2, 4, 2, 4, ..., 4, 1


Gaussian Quadrature

Legendre Polynomial Nodes and Weights

nNodes (xᵢ)Weights (wᵢ)Exact for degree
1021
2±1/√313
30, ±√(3/5)8/9, 5/95
4±0.339981, ±0.8611360.652145, 0.3478557

Transformation

To integrate over [a, b] instead of [−1, 1]: x = ((b−a)t + (b+a))/2, dx = (b−a)/2 dt


Examples


Theorems


Python Implementation

import numpy as np
from typing import Callable, Tuple


class NumericalIntegrator:
    """Comprehensive numerical integration toolkit."""

    @staticmethod
    def trapezoidal(f: Callable, a: float, b: float,
                    n: int = 1000) -> Tuple[float, float]:
        """Composite trapezoidal rule. Returns (result, error_estimate)."""
        x = np.linspace(a, b, n + 1)
        y = f(x)
        h = (b - a) / n
        result = h * (y[0] / 2 + np.sum(y[1:-1]) + y[-1] / 2)

        # Error estimate via Richardson extrapolation
        x2 = np.linspace(a, b, 2 * n + 1)
        y2 = f(x2)
        h2 = h / 2
        result2 = h2 * (y2[0] / 2 + np.sum(y2[1:-1]) + y2[-1] / 2)

        error = abs(result2 - result) / 3  # Richardson estimate
        return result, error

    @staticmethod
    def simpson(f: Callable, a: float, b: float,
                n: int = 1000) -> Tuple[float, float]:
        """Composite Simpson's rule (n forced even). Returns (result, error)."""
        if n % 2 != 0:
            n += 1
        x = np.linspace(a, b, n + 1)
        y = f(x)
        h = (b - a) / n
        result = h / 3 * (y[0] + 4 * np.sum(y[1::2]) +
                          2 * np.sum(y[2:-1:2]) + y[-1])

        # Error estimate
        n2 = 2 * n
        x2 = np.linspace(a, b, n2 + 1)
        y2 = f(x2)
        h2 = (b - a) / n2
        result2 = h2 / 3 * (y2[0] + 4 * np.sum(y2[1::2]) +
                            2 * np.sum(y2[2:-1:2]) + y2[-1])

        error = abs(result2 - result) / 15  # Richardson estimate
        return result, error

    @staticmethod
    def gauss_legendre(f: Callable, a: float, b: float,
                       n: int = 5) -> float:
        """n-point Gauss-Legendre quadrature."""
        nodes, weights = np.polynomial.legendre.leggauss(n)
        t = 0.5 * (b - a) * nodes + 0.5 * (b + a)
        w = 0.5 * (b - a) * weights
        return np.sum(w * f(t))

    @staticmethod
    def romberg(f: Callable, a: float, b: float,
                max_level: int = 10) -> np.ndarray:
        """Romberg integration using Richardson extrapolation."""
        R = np.zeros((max_level + 1, max_level + 1))

        # Trapezoidal estimates at various levels
        n = 1
        h = b - a
        R[0, 0] = h / 2 * (f(a) + f(b))

        for i in range(1, max_level + 1):
            n = 2 ** i
            h = (b - a) / n
            # Add new midpoints
            x_new = np.arange(1, n, 2) * h + a
            R[i, 0] = 0.5 * R[i-1, 0] + h * np.sum(f(x_new))

            # Richardson extrapolation
            for j in range(1, i + 1):
                R[i, j] = R[i, j-1] + (R[i, j-1] - R[i-1, j-1]) / (4**j - 1)

        return R

    @staticmethod
    def monte_carlo(f: Callable, domain: list,
                    n_samples: int = 100000) -> Tuple[float, float]:
        """Monte Carlo integration over a rectangular domain."""
        a, b = domain[0], domain[1]
        volume = b - a

        x = np.random.uniform(a, b, n_samples)
        values = f(x)
        result = volume * np.mean(values)
        std_err = volume * np.std(values) / np.sqrt(n_samples)

        return result, std_err


def demonstrate_integration():
    """Full demonstration of all integration methods."""
    print("=" * 60)
    print("NUMERICAL INTEGRATION METHOD COMPARISON")
    print("=" * 60)

    # Test 1: Smooth function
    f1 = lambda x: np.exp(-x**2)
    exact1 = np.sqrt(np.pi)  # ∫₋∞^∞ e^(-x²) dx = √π
    # But we integrate from -5 to 5 (close enough)

    from scipy.integrate import quad
    exact1_quad, _ = quad(f1, -5, 5)

    print(f"\n∫₋₅⁵ e^(-x²) dx")
    print(f"scipy quad: {exact1_quad:.12f}")

    integrator = NumericalIntegrator()

    for n in [10, 50, 200]:
        trap, err = integrator.trapezoidal(f1, -5, 5, n)
        simp, err = integrator.simpson(f1, -5, 5, n)
        print(f"  n={n:>3}: Trapezoidal {trap:.10f} (err {err:.2e}) | "
              f"Simpson {simp:.10f} (err {err:.2e})")

    # Gauss-Legendre
    for n in [3, 5, 10]:
        gl = integrator.gauss_legendre(f1, -5, 5, n)
        print(f"  Gauss-Legendre (n={n}): {gl:.10f}")

    # Romberg
    R = integrator.romberg(f1, -5, 5, max_level=6)
    print(f"\nRomberg integration:")
    for i in range(min(7, R.shape[0])):
        print(f"  Level {i}: {R[i, i]:.12f}")

    # Test 2: Oscillatory function
    print(f"\n{'=' * 60}")
    f2 = lambda x: np.sin(100 * x) / (1 + x**2)
    mc_result, mc_err = integrator.monte_carlo(f2, [-10, 10], 1_000_000)
    quad_result, _ = quad(f2, -10, 10)
    print(f"∫₋₁₀¹⁰ sin(100x)/(1+x²) dx")
    print(f"scipy quad:  {quad_result:.10f}")
    print(f"Monte Carlo: {mc_result:.10f} (±{mc_err:.2e})")


demonstrate_integration()

Applications in AI/ML

Variational Inference

In Bayesian neural networks, the posterior P(θ|D) is intractable. Variational inference approximates it with a tractable distribution q(θ) and minimizes KL(q||p), which requires computing integrals like E_q[log p(D|θ)] using quadrature or Monte Carlo.

Monte Carlo in ML

ApplicationMethodDescription
Bayesian inferenceMCMC, VISample from posterior distributions
Policy gradientsREINFORCEMonte Carlo estimation of expected returns
Variational autoencodersReparameterizationMonte Carlo gradient estimation
Normalizing flowsImportance samplingMonte Carlo estimation of likelihood
GANsMonte CarloEstimate partition function

Loss Function Integration

Some loss functions require integration: Expected Calibration Error (ECE) integrates over confidence bins, and proper scoring rules integrate over predicted distributions.


Common Mistakes

MistakeProblemSolution
Odd n for Simpson's ruleFormula requires even number of intervalsRound n up to nearest even number
Ignoring error estimateTrusting result without verificationAlways compute error estimate or compare with finer grid
Uniform spacing for peaked functionsWasted evaluations in flat regionsUse adaptive quadrature
Low N for Monte CarloSlow convergence (O(1/√N))Use variance reduction or quasi-Monte Carlo
Integrating over singularitiesInfinite errorTransform variable or split integral
Not checking integrabilityDivergent integral returns wrong valueVerify convergence before trusting result
Using Simpson's for oscillatoryO(h⁴) not enough for highly oscillatoryUse specialized oscillatory quadrature
Ignoring roundoff in long sumsKahan summation may be neededUse compensated summation for many terms

Interview Questions

Q1: Why is Gaussian quadrature more accurate than equally-spaced rules? A: Gaussian quadrature optimizes both node placement and weights to maximize the degree of exactness. With n nodes it integrates polynomials of degree 2n−1 exactly, while equally-spaced rules (Newton-Cotes) with n nodes achieve at most degree n (or n+1 for Simpson's). The trade-off is that Gaussian nodes are irrational, requiring more complex implementation.

Q2: When would you use Monte Carlo over deterministic quadrature? A: Monte Carlo is preferred for high-dimensional integrals (d > 4), where deterministic methods suffer the curse of dimensionality. It is also useful when the integrand is noisy, discontinuous, or only available through simulations. For low-dimensional smooth functions, deterministic methods are far more efficient.

Q3: How does adaptive quadrature decide where to subdivide? A: Adaptive quadrature estimates the local error on each subinterval. If the error exceeds the tolerance, the interval is split (usually in half). The process recurses until all subintervals meet the error tolerance. This concentrates evaluations where the integrand varies rapidly.

Q4: Explain the relationship between numerical integration and numerical differentiation. A: Numerical differentiation amplifies noise (it's an ill-conditioned problem), while numerical integration smooths it (it's well-conditioned). The error in numerical differentiation is O(ε/h) from roundoff plus O(h^p) from truncation, creating an optimal h ≈ √ε. Integration error simply decreases with h.

Q5: What is Richardson extrapolation and how is it used in Romberg integration? A: Richardson extrapolation combines numerical results at different step sizes to eliminate leading error terms. If R(h) = I + c₁h^p + ..., then combining R(h) and R(h/2) eliminates the h^p term, achieving higher accuracy. Romberg integration applies this recursively to trapezoidal rule estimates, achieving exponential convergence for smooth functions.

Q6: How do you handle integrable singularities? A: Strategies include: (1) variable substitution to remove the singularity (e.g., t = √x for x^(-1/2) singularity), (2) splitting the integral and handling the singular part analytically, (3) using Gauss-Jacobi or other specialized quadrature designed for singular weight functions, (4) adaptive quadrature which may concentrate points near the singularity.


Practice Problems


Quick Reference


Cross-References

  • 086-Floating-Point — Numerical integration error is bounded by floating point precision; long summations need Kahan summation
  • 087-Root-Finding — Adaptive quadrature uses error estimation similar to root-finding convergence criteria
  • 089-Interpolation — Quadrature rules are derived by integrating interpolating polynomials; Gaussian quadrature uses optimal node placement
  • 090-Error Analysis — Truncation error analysis determines the order of convergence for integration methods
  • Monte Carlo Methods — Integration is the foundation of Monte Carlo simulation in physics, finance, and ML
  • ODE Solvers: Every ODE solver (Euler, Runge-Kutta) uses quadrature internally to advance the solution

Need Expert Mathematics Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement