Functional Analysis
Core Definitions
Key Formulas
Important Theorems
Worked Examples
Practice Problems
Common Mistakes
| Mistake | Correct Approach |
|---|---|
| Assuming all normed spaces are complete | with absolute value is normed but not complete; completion gives |
| Confusing bounded and unbounded operators | Differentiation is unbounded; integration is bounded |
| Assuming the kernel trick requires explicit | The kernel computes without ever computing |
| Forgetting that RKHS elements are functions | An RKHS is a space of functions, not vectors in |
| Using the wrong inner product for | uses , not pointwise product |
| Confusing compact and bounded operators | All compact operators are bounded, but not conversely (e.g., identity on infinite-dim space is bounded but not compact) |
| Ignoring the regularization parameter | Without regularization (), the problem may be ill-posed if the kernel matrix is singular |
Connections to Machine Learning
Exam/Interview Questions
Q1: State and explain the Cauchy-Schwarz inequality. Why is it important?
Answer: , with equality iff and are linearly dependent. It provides an upper bound on the inner product in terms of norms, guarantees that the angle between vectors is well-defined (), and is used to prove the triangle inequality for norms induced by inner products.
Q2: What is the difference between a Banach space and a Hilbert space?
Answer: A Banach space has only a norm (distance structure), while a Hilbert space has an inner product (angle structure). Every Hilbert space is a Banach space, but not conversely. The key difference: Hilbert spaces satisfy the parallelogram law , which fails for general Banach spaces (e.g., , ). The inner product enables orthogonal projections, which are fundamental to many ML algorithms.
Q3: Explain the representer theorem and its significance for kernel methods.
Answer: Any minimizer of in an RKHS has the form . This reduces infinite-dimensional optimization to finding coefficients , where is the number of training points. It explains why kernel methods work: despite operating in potentially infinite-dimensional spaces, the solution is always a finite combination of kernel evaluations at training points.
Q4: Give an example of an unbounded operator and explain why boundedness matters.
Answer: The differentiation operator defined by is unbounded: while . Boundedness matters because bounded operators are continuous (preserve limits), and the spectral theorem applies to bounded self-adjoint operators. Unbounded operators require careful domain restrictions and are studied in quantum mechanics.
Q5: Why is a Hilbert space but is not?
Answer: has an inner product that induces the norm, and is complete. has a norm but no inner product that induces this norm (it fails the parallelogram law). While is a Banach space (complete), it is not a Hilbert space. This is why Fourier analysis works naturally in (orthonormal bases exist) but not in .
Q6: How does the spectral theorem relate to PCA?
Answer: PCA finds the eigenvectors of the covariance matrix . In the infinite-dimensional setting, the covariance operator defined by is a compact self-adjoint operator. The spectral theorem guarantees an orthonormal eigenbasis with eigenvalues . The top eigenfunctions capture the directions of maximum variance, generalizing PCA to function spaces (functional PCA).
Quick Reference
| Concept | Definition/Formula | Key Property |
|---|---|---|
| Norm | , , | Distance from origin |
| Inner Product | with conjugate symmetry, linearity | Enables geometry (angles) |
| Cauchy-Schwarz | Bound on inner product | |
| Hilbert Space | Complete inner product space | Orthonormal bases exist |
| Banach Space | Complete normed space | Convergence guaranteed |
| Bounded Operator | Continuous | |
| Adjoint | Generalizes transpose | |
| RKHS | Hilbert space of functions with bounded point evaluation | |
| Kernel Trick | Implicit high-dim computation | |
| Representer Theorem | Infinite-dim -> finite-dim | |
| Compact Operator | Maps bounded sets to precompact sets | "Almost finite-rank" |
| Spectral Theorem | Diagonalizes compact self-adjoint operators |
Cross-References
- Linear Algebra β Linear Algebra Basics: Finite-dimensional vector spaces
- Measure Theory β Advanced Measure Theory: Integration theory on function spaces
- Differential Geometry β Advanced Differential Geometry: Manifolds and differential forms
- Topology β Advanced Topological: Topological vector spaces
- Tensor Calculus β Advanced Tensor Calculus: Tensor products of vector spaces