Regularization: Ridge, Lasso and ElasticNet

The Problem of Overfitting

When we fit a model to training data, we want it to generalize well to unseen data. A model that memorizes training data but fails on test data is said to overfit.

Bias-Variance Tradeoff

The fundamental tension in supervised learning:

As model complexity increases:

• Bias decreases (model captures more patterns)
• Variance increases (model becomes more sensitive)
• Total error follows a U-shaped curve

Why Regularization?

Regularization addresses overfitting by penalizing model complexity. Instead of minimizing only the loss function, we minimize:

where:

• is the regularization strength (hyperparameter)
• is the regularization term (penalty function)
• are the model parameters

Key insight: Regularization trades training performance for generalization.

Ridge Regression (L2 Regularization)

Formulation

Ridge regression adds an L2 penalty (squared magnitude of coefficients):

Expanding the penalty term:

Closed-Form Solution

Taking the derivative and setting to zero:

Key properties:

• Always has a unique solution (matrix is always invertible for )
• Coefficients are shrunk toward zero but never exactly zero
• Equivalent to OLS with modified covariance matrix

Geometric Interpretation

The Ridge solution is where the elliptical contours of the OLS loss meet the circular L2 constraint. Since circles are smooth, the intersection rarely occurs exactly on an axis.

Lasso Regression (L1 Regularization)

Formulation

Lasso (Least Absolute Shrinkage and Selection Operator) adds an L1 penalty (absolute magnitude):

Expanding:

No Closed-Form Solution

Unlike Ridge, Lasso has no analytical solution due to the non-differentiable absolute value. We use:

• Coordinate descent
• Proximal gradient methods
• Subgradient methods

Sparsity Property

The L1 penalty induces sparsity — it drives some coefficients to exactly zero:

where is the soft-thresholding operator.

Why does L1 produce sparsity?

The L1 constraint region has corners on the axes. The elliptical contours of OLS loss are more likely to intersect at these corners, yielding solutions where some .

ElasticNet: Best of Both Worlds

Formulation

ElasticNet combines L1 and L2 penalties:

Using the mixing parameter :

Advantages over Lasso

1. Grouped selection: When features are correlated, Lasso selects one arbitrarily; ElasticNet selects the group
2. Smooth penalty: Differentiable everywhere (unlike L1)
3. More stable: Less sensitive to small changes in data
4. : Works when number of features exceeds samples

Coefficient Shrinkage Visualization

Regularization Path

The regularization path shows how coefficients change as varies:

Key Observations

1. Left side ( small): All features included, model close to OLS
2. Right side ( large): Most coefficients zero, simple model
3. Sparsity: Lasso drives coefficients to exactly zero sequentially
4. : Value that minimizes cross-validation error
5. : Largest within 1 SE of (sparser model)

Choosing Lambda: Cross-Validation

We select using k-fold cross-validation:

Common Selection Strategies

Implementation in Python

Basic Setup

Ridge Regression

Lasso Regression

ElasticNet

Visualizing the Regularization Path

Comparing Models

When to Use Each Method

Practical Guidelines

1. Always Standardize

2. Start with Cross-Validation

3. Examine the Regularization Path

4. Compare with OLS

Summary

Key Takeaways:

1. Regularization prevents overfitting by penalizing model complexity
2. Ridge shrinks coefficients but keeps all features
3. Lasso performs automatic feature selection via sparsity
4. ElasticNet combines both benefits, often the best choice
5. Always use cross-validation to select the regularization strength
6. Standardize features before applying regularization