Model Interpretability: SHAP, LIME and Feature Importance

Why Interpretability Matters

Modern ML models achieve remarkable predictive accuracy, but black-box predictions are insufficient when trust, debugging, and regulatory compliance are required.

The Interpretability Imperative

When Interpretability Is Critical

Interpretability Spectrum

Models range from fully interpretable (you can read the rules) to completely opaque (only inputs and outputs visible).

Intrinsic vs Post-Hoc Interpretability

Feature Importance

Permutation Feature Importance

Measures importance by randomly shuffling each feature and measuring the drop in model performance.

Algorithm:

1. Train model , compute baseline score
2. For each feature :
   • Create permuted dataset (column shuffled randomly)
   • Compute
3. Feature importance:

Theoretical Foundation:

Permutation importance approximates the expected performance drop:

where denotes with feature replaced by a random permutation.

Implementation

Feature Importance Comparison

Gini Importance (Mean Decrease in Impurity)

For tree-based models, Gini importance measures the total reduction of impurity (Gini or entropy) contributed by each feature across all trees:

where is the impurity decrease at node in tree .

SHAP (SHapley Additive exPlanations)

Game Theory Foundation

SHAP is grounded in cooperative game theory. Each feature is treated as a "player" in a game, and SHAP values compute the fair marginal contribution of each feature.

The Shapley value for feature is:

where:

• is the set of all features
• is a subset of features not including
• is the value function (model prediction using features in )
• is the weighting factor

Key SHAP Properties

SHAP Additive Explanation

The SHAP explanation model is:

where indicates whether feature is included, and .

TreeSHAP: Efficient Exact Computation

For tree ensembles, TreeSHAP computes exact Shapley values in time (not exponential):

where:

• = number of trees
• = number of leaves
• = maximum depth

TreeSHAP avoids enumerating all feature subsets by exploiting the recursive structure of trees.

SHAP Implementation

SHAP for Deep Learning (DeepSHAP / GradientSHAP)

For neural networks, approximate Shapley values using:

LIME (Local Interpretable Model-agnostic Explanations)

Core Idea

LIME explains individual predictions by fitting a local linear model around the instance of interest in the perturbed input space.

LIME Algorithm

Objective:

where:

• is the black-box model
• is an interpretable model (e.g., linear model)
• is a kernel measuring proximity to instance
• is a complexity penalty (e.g., number of features)

Step-by-step:

1. Perturb: Generate samples around by sampling from
2. Predict: Get for each perturbed sample
3. Weight: Compute weights
4. Fit: Train interpretable model on weighted dataset
5. Explain: Return coefficients of as local explanation

LIME Implementation

LIME for Images and Text

Partial Dependence Plots (PDP)

Mathematical Definition

The partial dependence function shows the marginal effect of feature on the prediction:

where denotes all features except .

The empirical estimate is:

Individual Conditional Expectation (ICE)

ICE curves extend PDP by showing the effect for each individual instance:

While PDP shows the average effect, ICE reveals heterogeneity in the effect across instances.

PDP and ICE Implementation

Practical Comparison: When to Use What

Decision Framework

Advanced: SHAP Interaction Values

SHAP can decompose effects into main effects and interaction effects:

where captures the interaction between features and .

Implementation: Complete Pipeline

Evaluation: Knowledge Check

Q1. What is the fundamental difference between permutation importance and SHAP-based importance?

Q2. Why does SHAP use Shapley values rather than simple marginal contributions?

Q3. When would LIME provide a better explanation than SHAP?

Q4. What does the width of ICE curves in a PDP reveal?

Q5. Prove that Shapley values satisfy the efficiency property: .

Key Takeaways

1. SHAP provides theoretically grounded explanations with consistency guarantees
2. LIME is faster for single-instance explanations but lacks global consistency
3. PDP/ICE reveals how features affect predictions on average and individually
4. Permutation importance is the simplest model-agnostic global method
5. TreeSHAP makes exact Shapley computation feasible for tree ensembles
6. Always validate explanations against domain knowledge – no method is perfect

Next: Hyperparameter Tuning – Learn systematic approaches to optimizing model performance through Bayesian optimization, grid search, and random search.