Regularization: Ridge, Lasso and ElasticNet

The Problem of Overfitting

When we fit a model to training data, we want it to generalize well to unseen data. A model that memorizes training data but fails on test data is said to overfit.

Bias-Variance Tradeoff

The fundamental tension in supervised learning:

As model complexity increases:

• Bias decreases (model captures more patterns)
• Variance increases (model becomes more sensitive)
• Total error follows a U-shaped curve

Why Regularization?

Regularization addresses overfitting by penalizing model complexity. Instead of minimizing only the loss function, we minimize:

where:

• $\lambda \geq 0$ is the regularization strength (hyperparameter)
• $\Omega(\mathbf{w})$ is the regularization term (penalty function)
• $\mathbf{w}$ are the model parameters

Key insight: Regularization trades training performance for generalization.

Ridge Regression (L2 Regularization)

Formulation

Ridge regression adds an L2 penalty (squared magnitude of coefficients):

Expanding the penalty term:

Closed-Form Solution

Taking the derivative and setting to zero:

Key properties:

• Always has a unique solution (matrix is always invertible for $\lambda > 0$)
• Coefficients are shrunk toward zero but never exactly zero
• Equivalent to OLS with modified covariance matrix

Geometric Interpretation

The Ridge solution is where the elliptical contours of the OLS loss meet the circular L2 constraint. Since circles are smooth, the intersection rarely occurs exactly on an axis.

Lasso Regression (L1 Regularization)

Formulation

Lasso (Least Absolute Shrinkage and Selection Operator) adds an L1 penalty (absolute magnitude):

Expanding:

No Closed-Form Solution

Unlike Ridge, Lasso has no analytical solution due to the non-differentiable absolute value. We use:

• Coordinate descent
• Proximal gradient methods
• Subgradient methods

Sparsity Property

The L1 penalty induces sparsity — it drives some coefficients to exactly zero:

where $S_j = \mathbf{x}_j^T(\mathbf{y} - \mathbf{X}_{-j}\mathbf{w}_{-j})$ is the soft-thresholding operator.

Why does L1 produce sparsity?

The L1 constraint region has corners on the axes. The elliptical contours of OLS loss are more likely to intersect at these corners, yielding solutions where some $w_j = 0$.

ElasticNet: Best of Both Worlds

Formulation

ElasticNet combines L1 and L2 penalties:

Using the mixing parameter $\alpha \in [0, 1]$:

Advantages over Lasso

1. Grouped selection: When features are correlated, Lasso selects one arbitrarily; ElasticNet selects the group
2. Smooth penalty: Differentiable everywhere (unlike L1)
3. More stable: Less sensitive to small changes in data
4. $p > n$: Works when number of features exceeds samples

Coefficient Shrinkage Visualization

Regularization Path

The regularization path shows how coefficients change as $\lambda$ varies:

Key Observations

1. Left side ($\lambda$ small): All features included, model close to OLS
2. Right side ($\lambda$ large): Most coefficients zero, simple model
3. Sparsity: Lasso drives coefficients to exactly zero sequentially
4. $\lambda_{\min}$: Value that minimizes cross-validation error
5. $\lambda_{1\text{se}}$: Largest $\lambda$ within 1 SE of $\lambda_{\min}$ (sparser model)

Choosing Lambda: Cross-Validation

We select $\lambda$ using k-fold cross-validation:

Common Selection Strategies

Implementation in Python

Basic Setup

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge, Lasso, ElasticNet, RidgeCV, LassoCV, ElasticNetCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.datasets import make_regression
from sklearn.metrics import mean_squared_error, r2_score

np.random.seed(42)
n_samples, n_features = 200, 50
X, y, true_coef = make_regression(
    n_samples=n_samples,
    n_features=n_features,
    n_informative=10,
    noise=20,
    coef=True,
    random_state=42
)

correlation_matrix = np.eye(n_features)
for i in range(n_features - 1):
    correlation_matrix[i, i+1] = 0.8
    correlation_matrix[i+1, i] = 0.8

L = np.linalg.cholesky(correlation_matrix)
X = X @ L.T

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Ridge Regression

ridge_cv = RidgeCV(
    alphas=np.logspace(-3, 3, 100),
    scoring='neg_mean_squared_error',
    cv=5
)
ridge_cv.fit(X_train_scaled, y_train)

print(f"Best alpha (Ridge): {ridge_cv.alpha_:.4f}")

y_pred_ridge = ridge_cv.predict(X_test_scaled)
print(f"Ridge R²: {r2_score(y_test, y_pred_ridge):.4f}")
print(f"Ridge MSE: {mean_squared_error(y_test, y_pred_ridge):.4f}")
print(f"Non-zero coefficients: {np.sum(ridge_cv.coef_ != 0)}/{n_features}")

Lasso Regression

lasso_cv = LassoCV(
    alphas=np.logspace(-3, 1, 100),
    cv=5,
    max_iter=10000,
    random_state=42
)
lasso_cv.fit(X_train_scaled, y_train)

print(f"Best alpha (Lasso): {lasso_cv.alpha_:.4f}")

y_pred_lasso = lasso_cv.predict(X_test_scaled)
print(f"Lasso R²: {r2_score(y_test, y_pred_lasso):.4f}")
print(f"Lasso MSE: {mean_squared_error(y_test, y_pred_lasso):.4f}")
print(f"Non-zero coefficients: {np.sum(lasso_cv.coef_ != 0)}/{n_features}")
print(f"Zero coefficients: {np.sum(lasso_cv.coef_ == 0)}/{n_features}")

selected_features = np.where(lasso_cv.coef_ != 0)[0]
print(f"Selected features: {selected_features}")

ElasticNet

elastic_cv = ElasticNetCV(
    l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 1.0],
    alphas=np.logspace(-3, 1, 100),
    cv=5,
    max_iter=10000,
    random_state=42
)
elastic_cv.fit(X_train_scaled, y_train)

print(f"Best alpha: {elastic_cv.alpha_:.4f}")
print(f"Best l1_ratio: {elastic_cv.l1_ratio_:.4f}")

y_pred_elastic = elastic_cv.predict(X_test_scaled)
print(f"ElasticNet R²: {r2_score(y_test, y_pred_elastic):.4f}")
print(f"ElasticNet MSE: {mean_squared_error(y_test, y_pred_elastic):.4f}")
print(f"Non-zero coefficients: {np.sum(elastic_cv.coef_ != 0)}/{n_features}")

Visualizing the Regularization Path

def plot_regularization_path(X, y, alphas):
    fig, axes = plt.subplots(1, 3, figsize=(15, 5))
    models = [
        ('Ridge', Ridge()),
        ('Lasso', Lasso()),
        ('ElasticNet', ElasticNet(l1_ratio=0.5))
    ]
    for ax, (name, model) in zip(axes, models):
        coefs = []
        for a in alphas:
            model.set_params(alpha=a)
            model.fit(X, y)
            coefs.append(model.coef_)
        coefs = np.array(coefs)
        for i in range(coefs.shape[1]):
            ax.plot(np.log10(alphas), coefs[:, i], linewidth=0.8)
        ax.set_xlabel('log10(alpha)')
        ax.set_ylabel('Coefficient Value')
        ax.set_title(f'{name} Path')
        ax.axhline(y=0, color='k', linestyle='--', linewidth=0.5)
        ax.grid(True, alpha=0.3)
    plt.tight_layout()
    plt.savefig('regularization_paths.png', dpi=150, bbox_inches='tight')
    plt.show()

alphas = np.logspace(-3, 3, 200)
plot_regularization_path(X_train_scaled, y_train, alphas)

Comparing Models

def compare_regularization(X_train, y_train, X_test, y_test):
    results = {}
    ridge = RidgeCV(
        alphas=np.logspace(-3, 3, 100),
        scoring='neg_mean_squared_error',
        cv=5
    )
    ridge.fit(X_train, y_train)
    results['Ridge'] = {
        'alpha': ridge.alpha_,
        'r2': r2_score(y_test, ridge.predict(X_test)),
        'mse': mean_squared_error(y_test, ridge.predict(X_test)),
        'n_nonzero': np.sum(ridge.coef_ != 0)
    }
    lasso = LassoCV(
        alphas=np.logspace(-3, 1, 100),
        cv=5,
        max_iter=10000,
        random_state=42
    )
    lasso.fit(X_train, y_train)
    results['Lasso'] = {
        'alpha': lasso.alpha_,
        'r2': r2_score(y_test, lasso.predict(X_test)),
        'mse': mean_squared_error(y_test, lasso.predict(X_test)),
        'n_nonzero': np.sum(lasso.coef_ != 0)
    }
    elastic = ElasticNetCV(
        l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 1.0],
        alphas=np.logspace(-3, 1, 100),
        cv=5,
        max_iter=10000,
        random_state=42
    )
    elastic.fit(X_train, y_train)
    results['ElasticNet'] = {
        'alpha': elastic.alpha_,
        'l1_ratio': elastic.l1_ratio_,
        'r2': r2_score(y_test, elastic.predict(X_test)),
        'mse': mean_squared_error(y_test, elastic.predict(X_test)),
        'n_nonzero': np.sum(elastic.coef_ != 0)
    }
    return results

results = compare_regularization(X_train_scaled, y_train, X_test_scaled, y_test)

for model, metrics in results.items():
    print(f"\n{model}:")
    print(f"  alpha = {metrics['alpha']:.4f}")
    if 'l1_ratio' in metrics:
        print(f"  l1_ratio = {metrics['l1_ratio']:.2f}")
    print(f"  R2 = {metrics['r2']:.4f}")
    print(f"  MSE = {metrics['mse']:.4f}")
    print(f"  Non-zero coefficients: {metrics['n_nonzero']}/{n_features}")

When to Use Each Method

Practical Guidelines

1. Always Standardize

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

2. Start with Cross-Validation

from sklearn.linear_model import ElasticNetCV

model = ElasticNetCV(
    l1_ratio=[0.1, 0.5, 0.7, 0.9, 0.95, 1.0],
    alphas=np.logspace(-4, 2, 50),
    cv=5,
    max_iter=10000
)
model.fit(X_train_scaled, y_train)

3. Examine the Regularization Path

from sklearn.linear_model import lasso_path

alphas, coefs, _ = lasso_path(X_train_scaled, y_train, alphas=np.logspace(-3, 1, 50))
n_nonzero = [np.sum(np.abs(c) > 1e-10) for c in coefs.T]

plt.figure(figsize=(8, 5))
plt.plot(np.log10(alphas), n_nonzero)
plt.xlabel('log10(alpha)')
plt.ylabel('Number of Non-zero Coefficients')
plt.title('Feature Selection vs Regularization Strength')
plt.grid(True, alpha=0.3)
plt.show()

4. Compare with OLS

from sklearn.linear_model import LinearRegression

ols = LinearRegression()
ols.fit(X_train_scaled, y_train)

print("Model Comparison:")
print(f"  OLS: R2 = {r2_score(y_test, ols.predict(X_test_scaled)):.4f}")
print(f"  Ridge: R2 = {r2_score(y_test, ridge_cv.predict(X_test_scaled)):.4f}")
print(f"  Lasso: R2 = {r2_score(y_test, lasso_cv.predict(X_test_scaled)):.4f}")
print(f"  ElasticNet: R2 = {r2_score(y_test, elastic_cv.predict(X_test_scaled)):.4f}")

Summary

Key Takeaways:

1. Regularization prevents overfitting by penalizing model complexity
2. Ridge shrinks coefficients but keeps all features
3. Lasso performs automatic feature selection via sparsity
4. ElasticNet combines both benefits, often the best choice
5. Always use cross-validation to select the regularization strength $\lambda$
6. Standardize features before applying regularization