Taylor Series and Approximations

Why It Matters

---

What is a Taylor Series

---

Maclaurin Series

A Maclaurin series is simply a Taylor series centered at $a = 0$. It is the most commonly used form because derivatives at zero are often easier to compute.

---

Common Taylor Expansions

Memorizing these standard expansions is essential — they appear constantly in calculus, differential equations, and numerical methods.

---

Convergence

Not every Taylor series converges to its original function. Understanding convergence is critical.

---

Taylor's Theorem with Remainder

---

Approximation Applications

Taylor series are used to compute function values efficiently in practice.

---

Polynomial Approximation

---

Python Implementation

Basic Taylor Approximation

import numpy as np

def taylor_exp(x, n_terms=10):
    """Approximate e^x using the Maclaurin series."""
    result = 0
    for k in range(n_terms):
        result += x**k / np.math.factorial(k)
    return result

# Compare with numpy
x = 1.0
print(f"Exact:  {np.exp(x):.12f}")
print(f"Taylor: {taylor_exp(x, n_terms=10):.12f}")

# Output:
# Exact:  2.718281828459
# Taylor: 2.718281801146

Vectorized Taylor Approximation

import numpy as np

def taylor_sin(x, n_terms=8):
    """Vectorized Maclaurin series for sin(x)."""
    result = np.zeros_like(x, dtype=float)
    for k in range(n_terms):
        sign = (-1)**k
        power = 2*k + 1
        result += sign * x**power / np.math.factorial(power)
    return result

x = np.array([0.1, 0.5, 1.0, 2.0])
print(f"Exact:    {np.sin(x)}")
print(f"Taylor:   {taylor_sin(x)}")
print(f"Error:    {np.abs(np.sin(x) - taylor_sin(x))}")

SymPy Symbolic Taylor Series

from sympy import symbols, series, sin, cos, exp, log, oo

x = symbols('x')

# Taylor series expansions
print("e^x:", series(exp(x), x, 0, n=6))
print("sin(x):", series(sin(x), x, 0, n=8))
print("cos(x):", series(cos(x), x, 0, n=8))
print("ln(1+x):", series(log(1+x), x, 0, n=6))

# Compute specific derivatives
from sympy import factorial
def taylor_coeff(f, x, a, n):
    """Get the nth Taylor coefficient at point a."""
    return f.diff(x, n).subs(x, a) / factorial(n)

print(f" coefficient of x^3 in e^x: {taylor_coeff(exp(x), x, 0, 3)}")

Error Analysis

import numpy as np
import matplotlib.pyplot as plt

def taylor_error_analysis():
    """Compare Taylor approximations of different orders."""
    x = np.linspace(-3, 3, 200)
    exact = np.exp(x)
    
    errors = {}
    for n in [3, 5, 7, 10]:
        approx = np.zeros_like(x)
        for k in range(n + 1):
            approx += x**k / np.math.factorial(k)
        errors[n] = np.abs(exact - approx)
    
    # Plot
    plt.figure(figsize=(10, 6))
    for n, err in errors.items():
        plt.semilogy(x, err, label=f'n={n}')
    plt.xlabel('x')
    plt.ylabel('Absolute Error')
    plt.title('Taylor Series Error for e^x')
    plt.legend()
    plt.grid(True)
    plt.show()

taylor_error_analysis()

---

Applications in AI/ML

Linearization of Neural Networks

Gradient Descent Analysis

Taylor series explain why gradient descent works:

• The gradient $\nabla f$ is the first-order Taylor approximation
• Newton's method uses the second-order Taylor expansion for faster convergence
• Momentum methods can be viewed as modifying the Taylor approximation landscape

Taylor Series in Loss Functions

Many loss functions are analyzed using Taylor expansions:

• Cross-entropy loss near zero predictions (gradient explosion)
• Huber loss as a smooth approximation of MAE
• Softmax stability analysis (subtracting the max is equivalent to scaling the Taylor series)

Automatic Differentiation Connection

Automatic differentiation (AD) computes exact derivatives — the same derivatives needed for Taylor series. Modern ML frameworks like PyTorch and JAX use AD to:

• Compute gradients for backpropagation
• Enable higher-order derivatives for Taylor-based optimization
• Implement the Taylor-mode AD for efficient computation of Taylor coefficients

---

Common Mistakes

---

Interview Questions

Question 1: Why does $e^x$ have the simplest Maclaurin series?

Answer: Because all derivatives of $e^x$ are $e^x$ itself, and $e^0 = 1$. So $f^{(n)}(0) = 1$ for all $n$, giving the clean series $\sum \frac{x^n}{n!}$. No alternating signs, no zeros — every term contributes equally.

Question 2: How would you approximate $\sqrt{1.1}$ without a calculator?

Answer: Use the binomial series $(1+x)^\alpha$ with $x = 0.1$ and $\alpha = 1/2$:

The exact value is $\approx 1.0488088...$, so 3 terms give 5 digits of accuracy.

Question 3: Why is the Taylor series for $\ln(1+x)$ restricted to $|x| < 1$?

Answer: The coefficients $c_n = \frac{(-1)^{n+1}}{n}$ satisfy $\lim_{n\to\infty} |c_n/c_{n+1}| = 1$, giving $R = 1$. At $x = 1$, the series $1 - 1/2 + 1/3 - \cdots$ converges (alternating harmonic series), but at $x = -1$ it becomes $-1 - 1/2 - 1/3 - \cdots$, which diverges (harmonic series).

Question 4: What's the relationship between Taylor series and Fourier series?

Answer: Both decompose functions into simpler basis functions:

• Taylor series use polynomial basis $\{1, x, x^2, \ldots\}$ — best for local approximation
• Fourier series use trigonometric basis $\{1, \cos(nx), \sin(nx)\}$ — best for periodic functions

Taylor series require differentiability; Fourier series work for discontinuous functions. In higher dimensions, Taylor series extend to multivariate polynomials, while Fourier series extend to the Fourier transform.

Question 5: How do Taylor series relate to the concept of analyticity?

Answer: A function is analytic at a point if its Taylor series converges to the function in some neighborhood. All polynomial, exponential, trigonometric, and rational functions (away from poles) are analytic. However, $f(x) = e^{-1/x^2}$ (with $f(0)=0$) is smooth but not analytic — its Taylor series at 0 is identically zero, yet the function is nonzero for $x \neq 0$.

Question 6: Can you use Taylor series to solve differential equations?

Answer: Yes — the power series method assumes $y(x) = \sum_{n=0}^{\infty} a_n x^n$, substitutes into the ODE, and solves for the coefficients recursively. For example, solving $y' = y$ with $y(0) = 1$ gives $a_n = 1/n!$, recovering $e^x$. This method works for linear ODEs with polynomial coefficients and is the basis for special functions like Bessel functions and Legendre polynomials.

---

Practice Problems

Problem 1: Derive the Series

Problem 2: Error Estimation

Problem 3: Non-Standard Expansion

Problem 4: Real-World Application

Problem 5: Series Manipulation

---

Quick Reference

---

Cross-References

• Sequences and Series — foundations for understanding convergence
• Power Series — the general framework for Taylor series
• Maclaurin Series — the special case at $a = 0$
• L'Hôpital's Rule — can be proved using Taylor expansions
• Newton's Method — uses first-order Taylor approximation for root finding
• Numerical Integration — Simpson's rule is based on quadratic Taylor approximation
• Fourier Series — alternative decomposition using trigonometric basis
• Differential Equations — power series method for solving ODEs
• Optimization — second-order methods use Hessian (second Taylor term)
• Automatic Differentiation — computes exact Taylor coefficients via the chain rule
• Neural Tangent Kernel — linearization of neural networks via first-order Taylor expansion