Kernel Methods: Reproducing Kernel Hilbert Spaces

Module: Machine Learning | Difficulty: Advanced

Reproducing Kernel Hilbert Space (RKHS)

is an RKHS if there exists such that:

Representer Theorem

The optimal solution lies in the span of kernel evaluations at training points.

Mercer's Theorem

where and are eigenfunctions.

Kernel Design Rules

1. Positive combination:
2. Product:
3. Exponential:

import numpy as np

class KernelRidgeRegression:
    def __init__(self, kernel='rbf', gamma=1.0, alpha=1.0):
        self.kernel = kernel; self.gamma = gamma; self.alpha = alpha
    def _kernel(self, X1, X2):
        if self.kernel == 'rbf':
            sq = np.sum(X1**2,1).reshape(-1,1) + np.sum(X2**2,1) - 2*X1@X2.T
            return np.exp(-self.gamma * sq)
        return X1 @ X2.T
    def fit(self, X, y):
        K = self._kernel(X, X)
        self.alpha_ = np.linalg.solve(K + self.alpha*np.eye(len(y)), y)
        self.X_train = X
    def predict(self, X):
        return self._kernel(X, self.X_train) @ self.alpha_

Research Insight: Neural tangent kernels (NTK) show that wide neural networks converge to kernel methods in the infinite width limit. The kernel is determined by the architecture and initialization, providing a bridge between kernel methods and deep learning.