Regularization Theory: Ridge, Lasso, and Beyond

Module: Machine Learning | Difficulty: Advanced

Ridge Regression (Tikhonov)

Lasso (L1 Regularization)

Elastic Net

Comparison

Effective Degrees of Freedom

where are singular values of .

import numpy as np

class RidgeRegression:
    def __init__(self, lambda_=1.0):
        self.lambda_ = lambda_
    def fit(self, X, y):
        p = X.shape[1]
        self.beta = np.linalg.solve(X.T @ X + self.lambda_ * np.eye(p), X.T @ y)
        return self
    def predict(self, X):
        return X @ self.beta
    def effective_df(self, X):
        svd = np.linalg.svd(X, compute_uv=False)
        return np.sum(svd**2 / (svd**2 + self.lambda_))

Research Insight: Lasso's inconsistency in variable selection arises because the penalty is not differentiable at zero. The bias introduced by Lasso is proportional to the true coefficient magnitude, leading to underestimated effects.