Partial Derivatives and Gradients

Partial Derivatives and Gradients

What is a Partial Derivative

Computing Partial Derivatives

To compute , treat all variables except as constants and apply the standard single-variable differentiation rules.

Higher-Order Partial Derivatives

Just as we can take derivatives of derivatives in single-variable calculus, we can take partial derivatives of partial derivatives.

The Gradient

Key Properties of the Gradient:

• Points in the direction of steepest ascent of
• The magnitude equals the rate of steepest change in that direction
• The gradient is perpendicular (orthogonal) to level sets (contour lines) of
• At a local maximum or minimum, (the zero vector)

Directional Derivative

The gradient tells you the slope along each coordinate axis, but what if you want the slope in an arbitrary direction?

Important Facts:

• is maximized when (i.e., is parallel to ). The maximum value is .
• is minimized when (i.e., is antiparallel to ). The minimum value is .
• when , meaning you are moving along a level curve.

Tangent Plane

Total Derivative

If and are both functions of a parameter , the chain rule for total derivatives gives:

Chain Rule for Partial Derivatives

Python Implementation

Numerical Gradient (Finite Differences)

import numpy as np

def numerical_gradient(f, x, h=1e-7):
    """Compute the gradient of f at x using central differences."""
    grad = np.zeros_like(x)
    for i in range(len(x)):
        x_plus = x.copy()
        x_minus = x.copy()
        x_plus[i] += h
        x_minus[i] -= h
        grad[i] = (f(x_plus) - f(x_minus)) / (2 * h)
    return grad

# Example: f(x, y) = x^2 + y^2, gradient should be [2x, 2y]
f = lambda x: x[0]**2 + x[1]**2
x = np.array([1.0, 2.0])
print(f"Numerical gradient at (1,2): {numerical_gradient(f, x)}")
# Output: [2.0, 4.0]

Symbolic Differentiation with SymPy

from sympy import symbols, diff, sin, exp, pprint

x, y = symbols('x y')

# Define a multivariable function
f = x**2 * y + sin(x * y)

# Compute partial derivatives
fx = diff(f, x)
fy = diff(f, y)

print(f"f(x,y)  = {f}")
print(f"df/dx   = {fx}")
print(f"df/dy   = {fy}")

# Second-order partials
fxx = diff(f, x, 2)
fxy = diff(f, x, y)
print(f"d2f/dx2 = {fxx}")
print(f"d2f/dxdy = {fxy}")

# Gradient at a point
grad_at = {x: 1, y: 2}
print(f"Gradient at (1,2): df/dx={fx.subs(grad_at)}, df/dy={fy.subs(grad_at)}")

Vectorized Gradient with NumPy

import numpy as np

def gradient_field(f, grid_x, grid_y, h=1e-7):
    """Compute gradient on a 2D grid."""
    dz_dx = (f(grid_x + h, grid_y) - f(grid_x - h, grid_y)) / (2 * h)
    dz_dy = (f(grid_x, grid_y + h) - f(grid_x, grid_y - h)) / (2 * h)
    return dz_dx, dz_dy

# Example: f(x,y) = sin(x) * cos(y)
f = lambda x, y: np.sin(x) * np.cos(y)
x = np.linspace(-np.pi, np.pi, 5)
y = np.linspace(-np.pi, np.pi, 5)
X, Y = np.meshgrid(x, y)

gx, gy = gradient_field(f, X, Y)
print("Gradient at (pi/4, pi/4):")
print(f"  df/dx = {np.cos(np.pi/4) * np.cos(np.pi/4):.4f}")
print(f"  df/dy = {-np.sin(np.pi/4) * np.sin(np.pi/4):.4f}")

Directional Derivative in Python

import numpy as np

def directional_derivative(f, point, direction, h=1e-7):
    """Compute directional derivative of f at point in given direction."""
    grad = np.zeros_like(point, dtype=float)
    for i in range(len(point)):
        p_plus = point.copy().astype(float)
        p_minus = point.copy().astype(float)
        p_plus[i] += h
        p_minus[i] -= h
        grad[i] = (f(p_plus) - f(p_minus)) / (2 * h)
    
    unit_dir = direction / np.linalg.norm(direction)
    return np.dot(grad, unit_dir), grad

# f(x,y) = x^2 - y^2 at (1,2) in direction (3,4)
f = lambda p: p[0]**2 - p[1]**2
point = np.array([1.0, 2.0])
direction = np.array([3.0, 4.0])

dd, grad = directional_derivative(f, point, direction)
print(f"Gradient: {grad}")       # [2, -4]
print(f"Directional derivative: {dd}")  # -2.0

Applications in AI/ML

Gradient Descent

The core training loop for neural networks:

Each component tells the optimizer how to adjust parameter to reduce the loss. The negative sign means we move in the direction of steepest descent.

Backpropagation

Backpropagation is the chain rule applied to a deep neural network. For a network with layers :

Each factor is a Jacobian matrix of partial derivatives. The gradient is computed by chaining these Jacobians from output to input — this is why the chain rule is the backbone of deep learning.

Hessian in Optimization

The matrix of second-order partials (the Hessian) determines the curvature of the loss surface:

• If is positive definite at a critical point, it is a local minimum
• Newton's method uses to converge faster than gradient descent
• In practice, is too large to compute for deep networks, so first-order methods (SGD, Adam) are used instead

Common Mistakes

Interview Questions

Q1: What does the gradient vector point toward?

Q2: When are mixed partial derivatives not equal?

Q3: Why is the gradient orthogonal to level curves?

Q4: How does gradient descent use partial derivatives in neural network training?

Q5: What is the difference between the directional derivative and the gradient?

Q6: Why can't we use the Hessian for large neural networks?

Practice Problems

Problem 1: Partial Derivatives

Problem 2: Gradient and Directional Derivative

Problem 3: Second-Order Partials and Clairaut's Theorem

Problem 4: Tangent Plane

Problem 5: Multivariable Chain Rule

Quick Reference

Cross-References

• Derivatives: Prerequisite — single-variable differentiation rules -> Derivatives and Differentiation
• Chain Rule: Single-variable chain rule and implicit differentiation -> Chain Rule and Implicit Differentiation
• Multivariable Calculus: Extrema, Lagrange multipliers, Jacobians -> Multivariable Calculus
• Optimization: Gradient descent, convergence, convexity -> Optimization Fundamentals
• Gradient Descent: Practical GD variants and learning rate schedules -> Gradient Descent
• Matrix Calculus: Derivatives of matrix-valued functions -> Matrix Calculus
• Taylor Series: Polynomial approximations using higher-order derivatives -> Taylor Series
• Newton's Method: Second-order optimization using the Hessian -> Newton's Method