Multivariable Calculus

Multivariable Functions

Example: A neural network layer with 256 inputs and 64 outputs is a vector function .

Partial Derivatives Review

Key Rules:

Sum Rule:
Product Rule:
Chain Rule:
Clairaut's Theorem: (for continuous second derivatives)

The Gradient Vector

Example: For :

At point :

Directional Derivative

Example: For at , direction :

, so

Hessian Matrix

Example: For :

At critical point :

Multivariable Taylor Series

Second-order approximation (scalar case):

Vector form: is a quadratic form.

Critical Points Classification

Classification rules for :

		Classification
		Local minimum
		Local maximum
—		Saddle point
—		Inconclusive

Example:

, eigenvalues local minimum

Double Integrals

Change of Variables

Common transformations:

Transformation
Polar
Cylindrical
Spherical

Python Implementation

import numpy as np
import sympy as sp

# ============================================
# 1. Partial Derivatives with SymPy
# ============================================
x, y = sp.symbols('x y')
f = x**2 * y + sp.sin(y)

# Partial derivatives
df_dx = sp.diff(f, x)
df_dy = sp.diff(f, y)
print(f"f = {f}")
print(f"∂f/∂x = {df_dx}")      # 2*x*y
print(f"∂f/∂y = {df_dy}")      # x**2 + cos(y)

# Gradient
grad_f = [df_dx, df_dy]
print(f"∇f = {grad_f}")

# ============================================
# 2. Hessian Matrix with SymPy
# ============================================
f_hess = x**4 + y**4 - 4*x*y + 1
H = sp.hessian(f_hess, (x, y))
print(f"\nHessian of {f_hess}:")
sp.pprint(H)

eigenvals = H.eigenvals()
print(f"Eigenvalues at (1,1): {H.subs({x:1, y:1}).eigenvals()}")

# ============================================
# 3. Gradient Descent Implementation
# ============================================
def gradient_descent(grad_f, x0, lr=0.01, max_iter=1000, tol=1e-8):
    """Simple gradient descent optimization."""
    x = np.array(x0, dtype=float)
    history = [x.copy()]
    
    for i in range(max_iter):
        g = np.array(grad_f(x), dtype=float)
        if np.linalg.norm(g) < tol:
            print(f"Converged in {i} iterations")
            break
        x = x - lr * g
        history.append(x.copy())
    
    return x, np.array(history)

# Minimize f(x,y) = x^2 + y^2 - 2x - 4y + 5
f = lambda v: v[0]**2 + v[1]**2 - 2*v[0] - 4*v[1] + 5
grad = lambda v: np.array([2*v[0]-2, 2*v[1]-4])

x_min, hist = gradient_descent(grad, [5.0, 5.0], lr=0.1)
print(f"\nMinimum at: {x_min}")  # Should be [1, 2]

# ============================================
# 4. Numerical Jacobian
# ============================================
def numerical_jacobian(f, x, h=1e-7):
    """Compute Jacobian matrix numerically using central differences."""
    n = len(x)
    m = len(f(x))
    J = np.zeros((m, n))
    for j in range(n):
        x_plus = x.copy()
        x_minus = x.copy()
        x_plus[j] += h
        x_minus[j] -= h
        J[:, j] = (f(x_plus) - f(x_minus)) / (2 * h)
    return J

# Example: f(x,y) = [x^2 + y, x*y^2]
f_vec = lambda v: np.array([v[0]**2 + v[1], v[0]*v[1]**2])
x_point = np.array([1.0, 2.0])
J = numerical_jacobian(f_vec, x_point)
print(f"\nJacobian at (1,2):\n{J}")
# Expected: [[2, 1], [4, 4]]

# ============================================
# 5. Numerical Hessian
# ============================================
def numerical_hessian(f, x, h=1e-5):
    """Compute Hessian matrix numerically."""
    n = len(x)
    H = np.zeros((n, n))
    f0 = f(x)
    
    for i in range(n):
        for j in range(n):
            x_pp = x.copy()
            x_pm = x.copy()
            x_mp = x.copy()
            x_mm = x.copy()
            
            x_pp[i] += h; x_pp[j] += h
            x_pm[i] += h; x_pm[j] -= h
            x_mp[i] -= h; x_mp[j] += h
            x_mm[i] -= h; x_mm[j] -= h
            
            H[i, j] = (f(x_pp) - f(x_pm) - f(x_mp) + f(x_mm)) / (4 * h**2)
    
    return H

# Test on f(x,y) = x^4 + y^4 - 4xy + 1
f_test = lambda v: v[0]**4 + v[1]**4 - 4*v[0]*v[1] + 1
H_num = numerical_hessian(f_test, np.array([1.0, 1.0]))
print(f"\nNumerical Hessian at (1,1):\n{H_num}")

# ============================================
# 6. Double Integration with SciPy
# ============================================
from scipy import integrate

def integrand(y, x):
    return np.exp(-(x**2 + y**2))

# Integrate over circle of radius R
result, error = integrate.dblquad(
    integrand,
    -2, 2,           # x bounds
    lambda x: -np.sqrt(4-x**2),  # y lower
    lambda x: np.sqrt(4-x**2)    # y upper
)
print(f"\nDouble integral over circle: {result:.6f}")
print(f"Expected (≈ π(1-e^(-4))): {np.pi*(1-np.exp(-4)):.6f}")

Applications in AI/ML

1. Gradient Descent: The most fundamental optimization algorithm uses . The gradient provides the direction of steepest descent.

2. Second-Order Methods: Newton's method uses the Hessian to achieve quadratic convergence:

The Hessian's eigenvalues determine convergence rate: eigenvalues close to zero slow convergence, while large condition numbers indicate ill-conditioning.

3. Natural Gradient: The Fisher information matrix (an expected Hessian) accounts for the geometry of the parameter space, providing better updates than standard gradient descent.

4. Saddle Points in High Dimensions: In high-dimensional loss landscapes, saddle points are exponentially more common than local minima. The Hessian's indefinite eigenvalues reveal saddle points that trap first-order methods.

5. Normalizing Flows: Use the change of variables formula with the Jacobian determinant to transform simple distributions into complex ones:

6. Backpropagation: Each layer's Jacobian matrix propagates gradients through the chain rule. Vanishing/exploding gradients correspond to Jacobian eigenvalues less than or greater than 1.

Common Mistakes

Mistake	Correct Approach
Forgetting chain rule:
Ignoring mixed partials	Verify for continuous functions
Wrong integration order	Check bounds: vs
Missing Jacobian determinant	Always include when changing variables
Confusing with	is a vector; is a differential (covector)
Not normalizing direction vector	must satisfy for directional derivative
Assuming Hessian is positive definite	Check eigenvalues, not just diagonal entries
Forgetting absolute value in Jacobian	Use for area/volume element

Interview Questions

Practice Problems

Quick Reference

Concept	Formula	Key Property
Gradient		Direction of steepest ascent
Directional Derivative	,	Rate of change in direction
Hessian		Symmetric matrix; determines curvature
Taylor (2nd order)		Quadratic approximation
Critical point		Candidate for min/max/saddle
Min	(all )	Convex curvature
Max	(all )	Concave curvature
Saddle	indefinite (mixed-sign eigenvalues)	Curves up in some directions, down in others
Double integral		Volume under surface
Jacobian		Local linear approximation of
Jacobian det.		Area/volume scaling factor
Polar coords	, ,	Use for circular regions

Cross-References

Linear Algebra: Eigenvalues and Eigenvectors — Essential for Hessian analysis and classification of critical points
Optimization: Calculus Optimization — Direct application of multivariable gradients
Probability: Probability Distributions — Joint distributions and expectations via double integrals
Calculus I: Calculus Derivatives — Foundation for partial derivatives and gradients
Calculus II: Calculus Integrals — Foundation for multiple integrals
Matrix Calculus: Matrix Calculus — Efficient computation of gradients in vectorized form

Multivariable Calculus