ML Foundations

Preventing Overfitting — Ridge, Lasso, and Elastic Net

Regularization constrains model complexity by adding penalty terms to the loss function, helping models generalize better to unseen data.

Ridge (L2) — shrinks weights toward zero to prevent overfitting when all features are potentially useful
Lasso (L1) — zeros out irrelevant features, performing automatic feature selection
Elastic Net — combines both penalties for the best of Ridge and Lasso

"Simplicity is the ultimate sophistication." — Leonardo da Vinci

Regularization — Complete Guide

Regularization prevents overfitting by adding a penalty term to the loss function, constraining model complexity.

The Problem

L1 vs L2 Constraint Regions

Architecture Diagram

Without regularization:
  Model fits training data perfectly
  Complex models with large weights
  High variance (overfitting)
  Poor generalization to new data

With regularization:
  Model balances fit and simplicity
  Smaller weights
  Lower variance (less overfitting)
  Better generalization

Ridge Regression (L2)

Coefficient Shrinkage Diagram

Lasso Regression (L1)

Elastic Net

Regularization Path

Python Implementation

Choosing Alpha

Architecture Diagram

α = 0: No regularization (original model)
α = ≡: All weights = 0 (trivial model)

Use cross-validation to find optimal α:
alphas = [0.001, 0.01, 0.1, 1, 10, 100]
for a in alphas:
    model = Ridge(alpha=a)
    score = cross_val_score(model, X, y, cv=5).mean()
    print(f"α={a}: {score:.3f}")

Key Takeaways

What to Learn Next

-> Linear Regression Understand the foundational model where Ridge and Lasso regularization are applied.

-> Logistic Regression Extend regularization to classification problems with penalized logistic models.

-> Model Evaluation Learn cross-validation techniques for selecting the optimal regularization strength.

-> Model Selection Compare algorithms and tune hyperparameters including regularization parameters.

-> Training Deep Networks Apply dropout, weight decay, and batch normalization as regularization in deep learning.

-> SVM Explore maximum margin classifiers that implicitly use L2 regularization.

Regularization — Ridge, Lasso and Elastic Net Complete Guide