Advanced Topics
LLM Interpretability
Understanding what happens inside the black box—mechanistic interpretability, probing, and activation patching to reverse-engineer language model cognition.
- Mechanistic — Reverse-engineering circuits and algorithms
- Probing — Training classifiers on internal representations
- Patching — Causal interventions on model activations
If you can't explain it simply, you don't understand it well enough.
LLM Interpretability
Understanding what happens inside the black box—mechanistic interpretability, probing, and activation patching to reverse-engineer language model cognition.
Levels of Interpretability
Interpretability operates at multiple levels of abstraction:
| Level | Question | Methods |
|---|---|---|
| Neuron | What does this neuron detect? | Activation visualization, maximally activating examples |
| Feature | What features does the model use? | Sparse autoencoders, dictionary learning |
| Circuit | What algorithm does this subnetwork implement? | Activation patching, causal intervention |
| Representation | What concepts are encoded? | Probing, geometric analysis |
| Behavior | Why does the model produce this output? | Attention visualization, Chain-of-Thought analysis |
Probing Methods
Linear Probing
The simplest probe: a linear classifier trained on internal representations:
A probe that achieves high accuracy suggests the information is linearly decodable from the representation—meaning the model has learned to encode it.
Non-Linear Probing
When linear probes fail, more expressive probes (MLPs) can detect non-linear representations:
What Probing Reveals
Research has revealed that LLMs encode rich internal representations:
| Layer Level | What Is Encoded | Example |
|---|---|---|
| Early layers | Token syntax, part-of-speech | Nouns vs verbs |
| Middle layers | Syntactic structure, semantics | Dependency relations |
| Late layers | Task-specific features, facts | Entity properties, relations |
Causal Probing
To establish that the model actually uses a representation, we must intervene causally:
Mechanistic Interpretability
Circuit Discovery
The core idea: identify a minimal subgraph of the model's computational graph that is sufficient to perform a specific task.
Activation Patching
Activation patching (also called causal tracing) is the primary tool for identifying circuits:
The procedure:
- Run the model on a clean input (produces correct output)
- Run on a corrupted input (produces incorrect output)
- For each component, replace its corrupted activation with the clean one
- Measure how much the output recovers toward correct
Types of Patching
There are several variants of activation patching, each targeting different computational structures:
The choice of patching granularity depends on the hypothesis being tested. Node patching identifies important components; edge patching identifies important information pathways.
Sparse Autoencoders
Large language models may encode concepts in superposed (polysemantic neurons). Sparse autoencoders decompose activations into interpretable features:
The sparsity constraint forces the autoencoder to learn a decomposed representation where each feature corresponds to a meaningful concept.
Practice Exercises
-
Conceptual: Explain the difference between interpretability by design (e.g., attention heads) and interpretability by analysis (e.g., probing). Why is causal intervention necessary?
-
Mathematical: If a linear probe achieves 95% accuracy on a binary classification task from layer 12 representations, and the random baseline is 50%, compute the probe's effective signal. Does this prove the model uses this information?
-
Practical: Using a small GPT-2 model, implement activation patching to identify which attention heads are responsible for in-context copying of a specific token.
-
Research: Compare the information-theoretic capacity of a superposed representation vs. an orthogonal one. How does the number of features that can be stored scale with dimensionality?
What to Learn Next
-> LLM Architecture Deep Dive Understanding transformers, attention, and internal computations.
-> Mixture of Experts Sparse architectures and conditional computation in LLMs.
-> LLM Watermarking Statistical watermarks and detection of AI-generated text.
-> Hallucination Detection Detecting factual errors through internal model analysis.
-> Environmental Impact of LLMs Energy costs and sustainable AI practices.
-> Future of LLMs Trends, predictions, and emerging capabilities.