AIC and BIC — Information Criteria for Model Selection

Information criteria balance model fit against complexity to select the best model among candidates. They provide a principled way to avoid overfitting.

Akaike Information Criterion (AIC)

Bayesian Information Criterion (BIC)

Corrected AIC (AICc)

For small samples, AIC can be overly liberal (overfits).

Deviance Information Criterion (DIC)

For Bayesian models:

Comparing Models

Likelihood Ratio Test

For nested models:

Information Criterion Comparison

| Metric | Values | Interpretation |

|--------|--------|---------------|

| | Best model | Strongest support |

| | | Substantial support |

| | | Considerable support |

| | | Much less support |

| | | Essentially no support |

Evidence Ratios

Python Implementation

import numpy as np

import pandas as pd

import statsmodels.api as sm

from scipy import stats

import matplotlib.pyplot as plt



np.random.seed(42)



# Generate data: true model is quadratic

n = 100

X = np.random.uniform(-3, 3, n)

Y = 2 + 1.5*X - 0.8*X**2 + np.random.randn(n) * 1.5



# Fit models of increasing complexity

models = {}

for degree in range(1, 6):

    X_poly = np.column_stack([X**i for i in range(degree + 1)])

    X_poly = sm.add_constant(X_poly)

    model = sm.OLS(Y, X_poly).fit()

    models[degree] = model



# Compare AIC and BIC

print("Model Comparison:")

print(f"{'Degree':<10} {'AIC':<10} {'BIC':<10} {'AICc':<10} {'k':<5}")

print("-" * 45)

for deg, m in models.items():

    aic = m.aic

    bic = m.bic

    k = m.df_model + 1

    aicc = aic + 2*k*(k+1)/(n - k - 1)

    print(f"{deg:<10} {aic:<10.1f} {bic:<10.1f} {aicc:<10.1f} {k:<5}")



# Akaike weights

aics = np.array([m.aic for m in models.values()])

delta_aics = aics - aics.min()

weights = np.exp(-delta_aics / 2)

weights = weights / weights.sum()



print("\nAkaike Weights:")

for deg, w in zip(models.keys(), weights):

    print(f"  Degree {deg}: {w:.3f}")



# Likelihood ratio test (nested models)

lr_stat = -2 * (models[1].llf - models[2].llf)

lr_pval = 1 - stats.chi2.cdf(lr_stat, 1)

print(f"\nLR test (degree 1 vs 2): ?²={lr_stat:.2f}, p={lr_pval:.4f}")

Worked Example

Key Takeaways

Related Topics

• See Cross-Validation for resampling-based model evaluation

• See ARIMA Models for using AIC/BIC in time series

• See ROC and AUC for classification model evaluation