Mechanistic Interpretability: Understanding Neural Networks from the Inside

Foundations, Research Methods, Practical Applications, Career Resilience and the Future of AI

Introduction

1. What Is This Book About?
2. Who Should Read This Book?
3. How This Book Is Structured
4. What This Book Does Not Claim
5. How to Use This Book

Chapter 1: What Is Mechanistic Interpretability?

1. The black box problem: why neural networks are hard to understand
2. Beyond behavioral evaluation: what mechanistic interpretability adds
3. How it differs from feature attribution, probing and explainable AI
4. The mechanistic explanation ideal: circuits, features and causal stories
5. Why this matters: safety, reliability, capability understanding and scientific knowledge

Chapter 2: The History and State of the Field

1. Early interpretability efforts: saliency maps, activation visualization and their limits
2. The transformer era and why mechanistic interpretability became tractable
3. The mechanistic turn (2020 onward): the transformer circuits agenda
4. Key laboratories, researchers and open source projects
5. Current research directions and how they relate

Chapter 3: Linear Algebra for Interpretability

1. High-dimensional spaces, vectors and dot products
2. Matrix factorization, SVD and dimensionality reduction
3. Eigendecomposition and its role in understanding representations
4. Orthogonality, projections and geometric intuition
5. Rank deficiency, shared directions and information bottlenecks

Chapter 4: Probability, Information and Optimization

1. Distributions over sequences and next-token prediction
2. Entropy, mutual information and representation compression
3. Cross-entropy loss and what it optimizes
4. Gradient descent, saddle points and landscape geometry
5. Implicit biases of optimization and their interpretability consequences

Chapter 5: Neural Network Architectures, From Perceptrons to Transformers

1. Feed-forward networks, backpropagation and the credit assignment problem
2. Residual connections and why they matter for interpretability
3. Transformer architecture: attention, MLP blocks, layer normalization, positional encoding
4. The residual stream as information highway
5. How tokens flow through a transformer: a concrete trace

Chapter 6: Representations and Embeddings

1. Token embeddings and what they capture
2. Learned representations across layers
3. Distributed representations and their properties
4. Linear structure in neural network representations
5. Feature spaces, distances and the geometry of learned embeddings

Chapter 7: The Problem of Polysemanticity

1. Monosemantic vs. polysemantic features
2. The overcompleteness problem: more features than dimensions
3. Examples of polysemantic neurons from early research
4. Why polysemanticity arises from optimization pressure
5. The implications for interpretability: why single-neuron analysis fails

Chapter 8: Superposition and Feature Geometry

1. What superposition is and isn’t
2. Sparse features, activation sparsity and the coding hypothesis
3. The geometric structure of superposed features
4. Eigenfeatures and principal components of activation space
5. The theoretical work of Elhage, Huben and others on superposition

Chapter 9: Sparse Autoencoders and Dictionary Learning

1. The sparse autoencoder architecture and how it works
2. Training objectives: reconstruction loss, L0/L1 penalties and dead feature problems
3. Key results: monosemantic features from SAEs
4. Variants: thresholded SAEs, JumpReLU SAEs, Gated SAEs, Transcoders
5. Practical considerations: dictionary sizes, feature quality metrics and current best practices

Chapter 10: Circuits and Computational Graphs

1. The circuit hypothesis: localized, composable mechanisms
2. Defining a circuit: inputs, outputs and intermediate nodes
3. How circuits differ from activation paths and attribution graphs
4. The relationship between features and circuits
5. The challenges of circuit discovery at scale

Chapter 11: Induction Heads and Algorithmic Behavior

1. What induction heads are: the discovery by Olsson et al.
2. The attention pattern: copying and sequence generalization
3. How induction heads implement in-context learning patterns
4. Empirical evidence and what we know about their prevalence
5. What this case teaches us about finding genuine mechanistic explanations

Chapter 12: Indirect Object Identification and Complex Reasoning Circuits

1. The IOI task: what models learn about indirect objects
2. The circuit structure: superpattern attention, name-mover heads, router heads
3. How evidence accumulated through patching and analysis
4. Why IOI matters: it shows circuits can explain relational reasoning
5. Limitations: IOI simplicity, task specificity and whether this generalizes

Chapter 13: Activation Analysis and Feature Visualization

1. Collecting activations: hooks, forward passes and data collection pipelines
2. Activation statistics: mean, variance, sparsity patterns across layers and components
3. Feature visualization: maximizing neuron activations, inversion and limitations
4. Logit lens and intermediate-token prediction
5. What conclusions each technique justifies and what they cannot

Chapter 14: Probing and Representation Analysis

1. Linear probes: what they test and what they measure
2. Probing tasks: syntactic, semantic, world knowledge and emergent properties
3. Probe performance vs. model use: the representational bottleneck hypothesis
4. Multiview probes and representation similarity analysis
5. When probes support mechanistic hypotheses and when they do not

Chapter 15: Activation Patching and Causal Tracing

1. Activation patching: concept, implementation and what it measures
2. Path patching and comparing clean vs corrupted inputs
3. Causal tracing: following influence through the network
4. Quantifying causal contributions: attribution scores and their interpretation
5. The assumptions and limitations: locality, additivity and intervention effects

Chapter 16: Ablation, Intervention and Counterfactual Testing

1. Ablation: zeroing, masking and disrupting specific components
2. Constructive interventions: injecting activations and testing effects
3. Counterfactual inputs: changing one aspect while holding others fixed
4. Gradient-based interventions and optimization-based attacks
5. How to design experiments that genuinely test a mechanistic hypothesis

Chapter 17: Decomposition and Attribution Techniques

1. Taylor decomposition and first-order attribution
2. Attention attribution: following information through attention patterns
3. Integrated gradients, attribution patches and related methods
4. Feature attribution vs. causal attribution: the difference that matters
5. Computational cost, interpretability of results and common failure modes

Chapter 18: Automated and Scalable Interpretability

1. Automated feature discovery: SAE scaling, neural gas and clustering
2. Automated circuit extraction: graph search, causal discovery algorithms and ML-based methods
3. The promise of automated interpretation at frontier scales
4. Current limitations and the human-in-the-loop requirement
5. The tension between automation and scientific understanding

Chapter 19: The Evidence We Have, Successful Explanations

1. Case studies where mechanistic explanations are robust and verified
2. Replication of published results and their generalizability
3. What successful explanations look like and how to recognize them
4. The gap between toy models and frontier models
5. Honest assessment: how far we have come

Chapter 20: Limitations and Pitfalls

1. Feature splitting, absorption and dictionary incompleteness
2. Misleading feature interpretations: confirmation bias and overfitting to anecdotes
3. Attribution errors: when patching gives wrong answers
4. Distribution shift and the instability of explanations
5. The curse of model scale: what we cannot do with current compute

Chapter 21: Beyond Mechanistic Interpretability, Complementary Approaches

1. Behavioral evaluation and its irreplaceable role
2. Formal methods, verification and their relationship to interpretability
3. Causal inference and its relevance
4. Scalable oversight and interpretability’s role in oversight
5. How to combine methods for stronger conclusions

Chapter 22: Interpretability for AI Safety

1. What interpretability can do now: model auditing, failure mode diagnosis, red teaming support
2. Speculative applications: deception detection, objective discovery, hidden capability monitoring
3. The gap between demonstrated capability and safety requirements
4. What interpretability alone cannot guarantee
5. The relationship to alignment strategies: oversight, training and deployment

Chapter 23: Governance, Auditing and Practical Deployment

1. Model cards, transparency reports and interpretability evidence
2. Regulatory expectations: what governments and regulators want
3. Closed-source models and the interpretability access problem
4. Practical auditing workflows using interpretability tools
5. The political economy of model transparency

Chapter 24: Learning Path, From Fundamentals to Research Competence

1. Prerequisites: what you need before starting (math, ML, programming)
2. Core reading: essential papers and resources in reading order
3. Tools to master: Python, PyTorch, TransformerLens, Hugging Face
4. Hands-on projects: progressively more sophisticated work
5. Research habits: how to think like an interpretability researcher

Chapter 25: Career Paths and Job Resilience

1. Where the jobs are: labs, safety orgs, startups, academia and adjacent roles
2. What employers value: research judgment, engineering, communication and domain knowledge
3. Building a portfolio that demonstrates real competence
4. Assessing opportunities: red flags, good signs and realistic expectations
5. Interview preparation and demonstrating research ability

Chapter 26: AI Agents and Career Resilience

1. What AI agents can already do: coding, literature review, basic analysis
2. What they cannot do well: causal reasoning, experimental design, scientific judgment
3. The changing labor economics of research and interpretation
4. Concrete strategies: transferable skills, complementary specializations, diversification
5. Scenario planning: what could change the field in the next 3-10 years

Chapter 27: Emerging Research Directions

1. Scalable feature discovery for billion- and trillion-parameter models
2. Interpretability of multimodal systems: vision, audio and cross-modal circuits
3. Recurrent, reasoning and planning architectures: new challenges
4. Causal representation learning and structured explanations
5. Integrating interpretability into the training loop

Chapter 28: Looking Ahead, What You Should Watch, Learn and Build

1. What to learn and build now: skills and projects with the highest expected value
2. What to prioritize in the next 1-3 years: key developments to follow
3. What to monitor in the next 3-5 years: structural changes that could reshape the field
4. Longer-term possibilities: scenarios that could make interpretability more or less relevant
5. Final assessment: is mechanistic interpretability worth the investment?

Conclusion

Technical Glossary

References