Derivatives and Differentiation
What is a Derivative
Basic Derivative Rules
The following rules allow us to differentiate complex functions by breaking them into simpler parts.
| Rule | Formula | Example |
|---|---|---|
| Constant | ||
| Constant Multiple | ||
| Power Rule | ||
| Sum Rule | ||
| Difference Rule | ||
| Product Rule | ||
| Quotient Rule | ||
| Chain Rule |
Derivatives of Common Functions
| Function | Derivative | Notes |
|---|---|---|
| Power rule, for any real | ||
| Its own derivative | ||
| General exponential | ||
| Logarithmic derivative | ||
| Change of base | ||
| Cyclic pattern | ||
| Note the negative sign | ||
| From quotient rule on | ||
| Inverse trig | ||
| Inverse trig | ||
| Power rule with | ||
| Sigmoid (ML critical) | ||
| Hyperbolic tangent |
Higher-Order Derivatives
| Notation | Meaning |
|---|---|
| First derivative β rate of change | |
| Second derivative β concavity / acceleration | |
| Third derivative β rate of change of acceleration | |
| -th derivative | |
| Leibniz notation for second derivative |
Implicit Differentiation
Logarithmic Differentiation
Mean Value Theorem
Applications
Related Rates
Optimization
Linear Approximation
Python Implementation
Symbolic Differentiation with SymPy
import sympy as sp
x = sp.Symbol('x')
# Symbolic derivatives
f = sp.ln(sp.sin(x))
print(f"Derivative of ln(sin(x)): {sp.diff(f, x)}")
g = x**2 * sp.exp(x)
print(f"Derivative of x^2 * e^x: {sp.diff(g, x)}")
# Higher-order derivatives
h = sp.sin(x)
print(f"Second derivative of sin(x): {sp.diff(h, x, 2)}")
print(f"Third derivative of sin(x): {sp.diff(h, x, 3)}")
# Partial derivatives (multivariate)
y = sp.Symbol('y')
f_multi = x**2 * y + sp.sin(x * y)
print(f"βf/βx = {sp.diff(f_multi, x)}")
print(f"βf/βy = {sp.diff(f_multi, y)}")
Numerical Differentiation
import numpy as np
def numerical_derivative(f, x, h=1e-7):
"""Central difference approximation."""
return (f(x + h) - f(x - h)) / (2 * h)
def numerical_second_derivative(f, x, h=1e-5):
"""Second derivative via finite differences."""
return (f(x + h) - 2 * f(x) + f(x - h)) / (h ** 2)
# Examples
f = np.sin
print(f"sin'(0.5) numerical: {numerical_derivative(f, 0.5):.6f}")
print(f"sin'(0.5) exact: {np.cos(0.5):.6f}")
g = lambda x: x**3 - 2*x + 1
print(f"g''(1) numerical: {numerical_second_derivative(g, 1):.6f}")
print(f"g''(1) exact: {6 * 1:.6f}")
Sigmoid and Its Derivative (ML)
import numpy as np
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def sigmoid_derivative(x):
s = sigmoid(x)
return s * (1 - s)
# Gradient check
x = 0.5
print(f"Sigmoid({x}) = {sigmoid(x):.6f}")
print(f"Sigmoid'({x}) = {sigmoid_derivative(x):.6f}")
# Verify numerically
h = 1e-7
numerical = (sigmoid(x + h) - sigmoid(x - h)) / (2 * h)
print(f"Numerical approx: {numerical:.6f}")
Applications in AI/ML
Gradient Descent
Chain Rule in Backpropagation
For a simple neural network layer , , loss :
import numpy as np
def sigmoid(x):
return 1 / (1 + np.exp(-x))
# Forward pass
x = np.array([1.0, 2.0])
W = np.array([[0.5, 0.3], [0.2, 0.7]])
b = np.array([0.1, 0.1])
z = W @ x + b
a = sigmoid(z)
# Backward pass via chain rule
y_true = np.array([1.0, 0.0])
dL_da = a - y_true # βL/βa
da_dz = a * (1 - a) # βa/βz (sigmoid derivative)
dL_dz = dL_da * da_dz # βL/βz
dL_dW = np.outer(dL_dz, x) # βL/βW
dL_db = dL_dz # βL/βb
Rate of Change in Feature Engineering
- Velocity: First derivative of position w.r.t. time.
- Acceleration: Second derivative.
- Jerk: Third derivative.
- In time-series ML, these derivatives (computed numerically) become features.
Common Mistakes
| Mistake | Incorrect | Correct | Explanation |
|---|---|---|---|
| Forgetting chain rule | Must multiply by inner derivative | ||
| Product rule order | Product rule is NOT just product of derivatives | ||
| Quotient rule sign | Numerator uses subtraction | ||
| Power rule on | Exponential rule differs from power rule | ||
| derivative | Common, but requires chain rule | ||
| Derivative of constant | Constants have zero rate of change |
Interview Questions
Q1: What is the derivative of ?
Q2: Explain the product rule in plain English. When do you use it?
Q3: Why is the sigmoid derivative so important in neural networks?
Q4: What is the difference between and ?
Q5: Prove that if for all in an interval, then is constant.
Q6: What are the conditions for a critical point to be a local minimum?
Practice Problems
Problem 1: Chain Rule
Problem 2: Product Rule
Problem 3: Implicit Differentiation
Problem 4: Optimization
Problem 5: Logarithmic Differentiation
Quick Reference
Cross-References
- Limits: Understanding limits is prerequisite for the derivative definition -> Limits and Continuity
- Chain Rule: In-depth coverage of the chain rule and implicit differentiation -> Chain Rule and Implicit Differentiation
- Partial Derivatives: Derivatives with multiple variables -> Partial Derivatives
- Multivariable Calculus: Gradients, Jacobians, and Hessians -> Multivariable Calculus
- Taylor Series: Polynomial approximations using derivatives -> Taylor Series
- Optimization: Gradient descent and convex optimization -> Optimization Fundamentals
- Linear Algebra: Matrix calculus for neural network derivatives -> Matrix Calculus
- Numerical Methods: Numerical differentiation and integration -> Numerical Methods