Model Interpretability & Explainability

1. The Interpretability Framework

Modern neural networks are black boxes — they achieve high accuracy but lack transparency. Interpretability methods aim to answer:

Why did the model make this prediction? (Local)
How does the model work? (Global)
What does the model learn? (Mechanistic)

1.1 Taxonomy of Interpretability Methods

Category	Method	Scope	Post-hoc
Model-specific	Attention visualization	Local	No
Model-agnostic	LIME, SHAP	Local	Yes
Mechanistic	Probing, circuit analysis	Global	Yes
Causal	Counterfactual explanations	Local	Yes

2. SHAP (SHapley Additive exPlanations)

2.1 Shapley Values from Game Theory

In cooperative game theory, the Shapley value assigns a value to player based on their marginal contribution:

where:

is the set of all players
is a coalition not containing player
is the value of coalition

2.2 SHAP in Machine Learning

For a model and input , define the value function:

where are the features in and the expectation is over the marginal distribution of the other features.

The SHAP value for feature :

2.3 SHAP Axioms

SHAP values satisfy four desirable properties:

Efficiency:
Symmetry: If for all , then
Null player: If for all , then
Linearity:

3. SHAP Force Plot Concept

<svg viewBox="0 0 800 400" xmlns="http://www.w3.org/2000/svg">
  <rect width="800" height="400" fill="#f8fafc"/>
  
  {/* Title */}
  <text x="400" y="30" text-anchor="middle" fill="#f8fafc" font-family="monospace" font-size="16" font-weight="bold">SHAP Values & Force Plot</text>
  
  {/* Feature contributions */}
  <g transform="translate(50, 50)">
    <text x="350" y="0" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="12">Feature Contributions to Prediction</text>
    
    {/* Base value */}
    <line x1="0" y1="40" x2="700" y2="40" stroke="#64748b" stroke-width="1" stroke-dasharray="5,3"/>
    <text x="-10" y="44" text-anchor="end" fill="#64748b" font-family="monospace" font-size="10">Base: E[f(x)]</text>
    
    {/* Feature bars */}
    {/* Age: positive contribution */}
    <rect x="350" y="60" width="120" height="30" fill="#3b82f6"/>
    <text x="410" y="80" text-anchor="middle" fill="white" font-family="monospace" font-size="9">Age: +0.35</text>
    <text x="345" y="80" text-anchor="end" fill="#60a5fa" font-family="monospace" font-size="10">Age=45</text>
    
    {/* Income: positive contribution */}
    <rect x="470" y="100" width="80" height="30" fill="#22c55e"/>
    <text x="510" y="120" text-anchor="middle" fill="white" font-family="monospace" font-size="9">Income: +0.25</text>
    <text x="465" y="120" text-anchor="end" fill="#22c55e" font-family="monospace" font-size="10">Income=80k</text>
    
    {/* Credit Score: positive contribution */}
    <rect x="550" y="140" width="60" height="30" fill="#22c55e"/>
    <text x="580" y="160" text-anchor="middle" fill="white" font-family="monospace" font-size="9">Credit: +0.15</text>
    <text x="545" y="160" text-anchor="end" fill="#22c55e" font-family="monospace" font-size="10">Score=720</text>
    
    {/* Debt: negative contribution */}
    <rect x="200" y="180" width="100" height="30" fill="#ec4899"/>
    <text x="250" y="200" text-anchor="middle" fill="white" font-family="monospace" font-size="9">Debt: -0.40</text>
    <text x="305" y="200" fill="#ec4899" font-family="monospace" font-size="10">Debt=50k</text>
    
    {/* Employment: positive contribution */}
    <rect x="610" y="220" width="40" height="30" fill="#3b82f6"/>
    <text x="630" y="240" text-anchor="middle" fill="white" font-family="monospace" font-size="9">+0.10</text>
    <text x="605" y="240" text-anchor="end" fill="#60a5fa" font-family="monospace" font-size="10">Emp=10y</text>
    
    {/* Final prediction arrow */}
    <line x1="650" y1="40" x2="650" y2="260" stroke="#f59e0b" stroke-width="3"/>
    <text x="660" y="150" fill="#fbbf24" font-family="monospace" font-size="11" font-weight="bold">f(x)=0.75</text>
  </g>
  
  {/* Shapley value computation */}
  <g transform="translate(50, 280)">
    <rect width="320" height="100" rx="8" fill="#f8fafc" stroke="#334155"/>
    <text x="160" y="20" text-anchor="middle" fill="#60a5fa" font-family="monospace" font-size="11" font-weight="bold">Shapley Value Formula</text>
    <text x="10" y="45" fill="#f8fafc" font-family="monospace" font-size="9">φi = Σ S⊆N\{i} [|S|!(|N|-|S|-1)!/|N|!]</text>
    <text x="10" y="65" fill="#f8fafc" font-family="monospace" font-size="9">    × [E[f(x)|xS,xi] - E[f(x)|xS]]</text>
    <text x="10" y="85" fill="#22c55e" font-family="monospace" font-size="9">Properties: Efficiency, Symmetry, Linearity</text>
  </g>
  
  {/* Comparison */}
  <g transform="translate(400, 280)">
    <rect width="350" height="100" rx="8" fill="#f8fafc" stroke="#334155"/>
    <text x="175" y="20" text-anchor="middle" fill="#a855f7" font-family="monospace" font-size="11" font-weight="bold">SHAP vs LIME</text>
    <text x="10" y="45" fill="#94a3b8" font-family="monospace" font-size="9">SHAP: Game-theoretic, exact (slow)</text>
    <text x="10" y="65" fill="#94a3b8" font-family="monospace" font-size="9">LIME: Local linear approx (fast)</text>
    <text x="10" y="85" fill="#94a3b8" font-family="monospace" font-size="9">Both: Model-agnostic, local explanations</text>
  </g>
</svg>

4. LIME (Local Interpretable Model-agnostic Explanations)

4.1 Local Linear Approximation

For a specific instance , LIME fits a simple model in the neighborhood:

where:

is the class of simple models (e.g., linear models)
is the proximity kernel
is the complexity penalty

4.2 Sampling and Weighting

Generate perturbed samples around
Obtain predictions from the black-box model
Compute weights
Fit weighted linear model:

4.3 LIME for Text and Images

Text LIME: Perturb by removing words or phrases.

Image LIME: Perturb by superpixel segmentation and random colorization.

5. Attention Visualization

5.1 Attention Rollout

For a Transformer with layers, the attention rollout computes the accumulated attention:

where is the attention matrix at layer .

5.2 Gradient-Weighted Attention

5.3 Attention vs. Gradient Methods

Method	Type	Faithfulness	Speed
Raw attention	Attention	Low	Fast
Attention rollout	Attention	Medium	Fast
Integrated gradients	Gradient	High	Slow
SHAP	Sampling	High	Very slow

6. Attention Rollout Diagram

<svg viewBox="0 0 800 400" xmlns="http://www.w3.org/2000/svg">
  <rect width="800" height="400" fill="#f8fafc"/>
  
  {/* Title */}
  <text x="400" y="30" text-anchor="middle" fill="#f8fafc" font-family="monospace" font-size="16" font-weight="bold">Attention Rollout Visualization</text>
  
  {/* Transformer layers */}
  <g transform="translate(50, 60)">
    {/* Layer 1 */}
    <text x="80" y="0" text-anchor="middle" fill="#60a5fa" font-family="monospace" font-size="11">Layer 1</text>
    <g transform="translate(30, 10)">
      <rect width="100" height="60" rx="5" fill="#f8fafc" stroke="#3b82f6" stroke-width="2"/>
      <text x="50" y="25" text-anchor="middle" fill="#60a5fa" font-family="monospace" font-size="9">A⁽¹⁾</text>
      <text x="50" y="45" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="8">Self-Attn</text>
    </g>
    
    {/* Layer 2 */}
    <text x="230" y="0" text-anchor="middle" fill="#a855f7" font-family="monospace" font-size="11">Layer 2</text>
    <g transform="translate(180, 10)">
      <rect width="100" height="60" rx="5" fill="#f8fafc" stroke="#a855f7" stroke-width="2"/>
      <text x="50" y="25" text-anchor="middle" fill="#c084fc" font-family="monospace" font-size="9">A⁽²⁾</text>
      <text x="50" y="45" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="8">Self-Attn</text>
    </g>
    
    {/* Layer 3 */}
    <text x="380" y="0" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="11">Layer 3</text>
    <g transform="translate(330, 10)">
      <rect width="100" height="60" rx="5" fill="#f8fafc" stroke="#22c55e" stroke-width="2"/>
      <text x="50" y="25" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="9">A⁽³⁾</text>
      <text x="50" y="45" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="8">Self-Attn</text>
    </g>
    
    {/* Final */}
    <text x="530" y="0" text-anchor="middle" fill="#f59e0b" font-family="monospace" font-size="11">Output</text>
    <g transform="translate(480, 10)">
      <rect width="100" height="60" rx="5" fill="#f8fafc" stroke="#f59e0b" stroke-width="2"/>
      <text x="50" y="25" text-anchor="middle" fill="#fbbf24" font-family="monospace" font-size="9">MLP</text>
      <text x="50" y="45" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="8">Classification</text>
    </g>
    
    {/* Arrows */}
    <line x1="130" y1="40" x2="180" y2="40" stroke="#64748b" stroke-width="2" marker-end="url(#1d77_arrowGray)"/>
    <line x1="280" y1="40" x2="330" y2="40" stroke="#64748b" stroke-width="2" marker-end="url(#1d77_arrowGray)"/>
    <line x1="430" y1="40" x2="480" y2="40" stroke="#64748b" stroke-width="2" marker-end="url(#1d77_arrowGray)"/>
  </g>
  
  {/* Attention matrices visualization */}
  <g transform="translate(50, 150)">
    <text x="350" y="0" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="11">Attention Rollout Formula</text>
    
    {/* Formula */}
    <g transform="translate(50, 20)">
      <rect width="600" height="80" rx="8" fill="#f8fafc" stroke="#334155"/>
      <text x="300" y="25" text-anchor="middle" fill="#f8fafc" font-family="monospace" font-size="12">A_rollout = Πₗ₌₁ᴸ (1/n · 11ᵀ + A⁽ˡ⁾)</text>
      <text x="300" y="50" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="10">Accounts for residual connections and layer-wise attention</text>
      <text x="300" y="70" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="10">Final: Attribution_i = Σⱼ A_rollout[i,j]</text>
    </g>
  </g>
  
  {/* Example attention flow */}
  <g transform="translate(50, 260)">
    <text x="350" y="0" text-anchor="middle" fill="#94a3b8" font-family="monospace" font-size="11">Example: Token Attribution</text>
    
    {/* Tokens */}
    <rect x="50" y="20" width="60" height="30" rx="4" fill="#3b82f6"/>
    <text x="80" y="40" text-anchor="middle" fill="white" font-family="monospace" font-size="9">[CLS]</text>
    
    <rect x="120" y="20" width="60" height="30" rx="4" fill="#3b82f6"/>
    <text x="150" y="40" text-anchor="middle" fill="white" font-family="monospace" font-size="9">The</text>
    
    <rect x="190" y="20" width="60" height="30" rx="4" fill="#a855f7"/>
    <text x="220" y="40" text-anchor="middle" fill="white" font-family="monospace" font-size="9">cat</text>
    
    <rect x="260" y="20" width="60" height="30" rx="4" fill="#22c55e"/>
    <text x="290" y="40" text-anchor="middle" fill="white" font-family="monospace" font-size="9">sat</text>
    
    <rect x="330" y="20" width="60" height="30" rx="4" fill="#3b82f6"/>
    <text x="360" y="40" text-anchor="middle" fill="white" font-family="monospace" font-size="9">on</text>
    
    <rect x="400" y="20" width="60" height="30" rx="4" fill="#f59e0b"/>
    <text x="430" y="40" text-anchor="middle" fill="white" font-family="monospace" font-size="9">mat</text>
    
    {/* Attribution scores */}
    <text x="80" y="75" text-anchor="middle" fill="#60a5fa" font-family="monospace" font-size="9">0.05</text>
    <text x="150" y="75" text-anchor="middle" fill="#60a5fa" font-family="monospace" font-size="9">0.10</text>
    <text x="220" y="75" text-anchor="middle" fill="#c084fc" font-family="monospace" font-size="9">0.35</text>
    <text x="290" y="75" text-anchor="middle" fill="#22c55e" font-family="monospace" font-size="9">0.20</text>
    <text x="360" y="75" text-anchor="middle" fill="#60a5fa" font-family="monospace" font-size="9">0.10</text>
    <text x="430" y="75" text-anchor="middle" fill="#fbbf24" font-family="monospace" font-size="9">0.20</text>
    
    {/* Color legend */}
    <text x="520" y="30" fill="#94a3b8" font-family="monospace" font-size="9">Attribution:</text>
    <rect x="520" y="40" width="15" height="15" fill="#3b82f6"/>
    <text x="540" y="52" fill="#94a3b8" font-family="monospace" font-size="8">Low</text>
    <rect x="520" y="60" width="15" height="15" fill="#a855f7"/>
    <text x="540" y="72" fill="#94a3b8" font-family="monospace" font-size="8">Medium</text>
    <rect x="520" y="80" width="15" height="15" fill="#f59e0b"/>
    <text x="540" y="92" fill="#94a3b8" font-family="monospace" font-size="8">High</text>
  </g>
  
  <defs>
    <marker id="1d77_arrowGray" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
      <polygon points="0 0, 10 3.5, 0 7" fill="#64748b"/>
    </marker>
  </defs>
</svg>

7. Saliency Maps

7.1 Vanilla Gradient

The simplest saliency method:

7.2 Integrated Gradients (Sundararajan et al., 2017)

where is a baseline input (e.g., black image or zero vector).

Axioms satisfied:

Completeness:
Sensitivity: If a feature matters, it gets non-zero attribution
Implementation invariance: Same function → same attribution

7.3 SmoothGrad

7.4 DeepLIFT

where is computed using a rescale rule that propagates contribution through the network.

8. Concept-Based Explanations

8.1 Testing with Concept Activation Vectors (TCAV)

Given a concept (e.g., "striped texture"):

Train a linear classifier in activation space to separate concept from random
Compute Concept Activation Vector:
Compute TCAV score:

8.2 Concept Bottleneck Models

The model first predicts concepts, then maps concepts to the output.

9. Causal Explanations

9.1 Counterfactual Explanations

Find the minimal change to the input that changes the prediction:

9.2 Wachter Counterfactual

where controls the trade-off between proximity and prediction match.

9.3 Causal Feature Attribution

Using structural causal models:

10. Mechanistic Interpretability

10.1 Circuit Analysis

Circuits are subnetworks that implement specific algorithms:

Induction heads: Implement in-context learning
Indirect object identification: Track coreference
Modular arithmetic: Implement Fourier transforms

10.2 Superposition Hypothesis

Neural networks represent more features than dimensions via superposition:

Features are encoded as near-orthogonal vectors in activation space.

10.3 Sparse Autoencoders for Interpretability

Train with sparsity penalty:

Each dimension of corresponds to an interpretable feature.

11. Evaluation of Explanations

11.1 Faithfulness

11.2 Robustness

An explanation is robust if:

11.3 Stability vs. Faithfulness Trade-off

There is often a tension between:

Stability: Similar inputs → similar explanations
Faithfulness: Explanations accurately reflect model behavior

Interpretability methods provide different trade-offs between these objectives, and the choice depends on the specific use case and requirements.

Model Interpretability & Explainable AI