The Ghost in the Weights

Approaches and Tools to Understand the Mechanistic Interpretability of LLMs

Introduction: Opening the Black Box

1. What Mechanistic Interpretability Is and Is Not
2. The Epistemological Stakes
3. What You Will Learn
4. How to Read This Book
5. A Note on Scope and Limitations

Chapter 1: The Promise of Mechanistic Understanding

1. Why Black-Box Models Fail Us
2. Mechanistic Versus Behavioral Interpretability
3. Landmark Discoveries
4. The Map of This Book
5. Open Questions and Honest Uncertainties

Chapter 2: Mathematical Foundations: Linear Algebra and Tensor Calculus

1. Vector Spaces and Inner Products
2. Basic Definitions
3. Why Geometry Matters
4. Subspaces and Projections
5. A Worked Example
6. Code: Visualizing Vector Geometry
7. Matrix Factorizations and Spectral Decomposition
8. Singular Value Decomposition
9. Why SVD Matters for Interpretability
10. Eigendecomposition
11. A Worked Example: Analyzing Attention with SVD
12. Tensor Algebra and Einstein Notation
13. The Convention
14. Why Einstein Notation for Transformers
15. Code: Verifying Einstein Notation
16. Position and Head Indices
17. Jacobian and Hessian Analysis
18. The Jacobian
19. The Hessian
20. Connection to Interpretability
21. Numerical Stability and Floating-Point Arithmetic
22. Floating-Point Basics
23. Attention-Specific Issues
24. Mixed Precision Training
25. Code: Numerical Stability in Practice
26. Summary

Chapter 3: Probability, Information Theory and Optimization

1. Probability Distributions Over Sequences
2. The Language Modeling Objective
3. Cross-Entropy Loss
4. Perplexity
5. Conditional Distributions and Next-Token Prediction
6. Code: Computing Language Model Probabilities
7. Entropy, Cross-Entropy and KL Divergence
8. Entropy
9. KL Divergence
10. Cross-Entropy and Its Decomposition
11. Practical Considerations
12. Gradient-Based Optimization
13. Gradient Descent
14. Stochastic Gradient Descent
15. Adam Optimizer
16. Learning Rate Schedules
17. Code: Understanding Adam Dynamics
18. Loss Landscapes and Critical Points
19. Local Minima and Saddle Points
20. Flat vs. Sharp Minima
21. Code: Visualizing Loss Landscape Curvature
22. Regularization and Generalization
23. Explicit Regularization
24. Implicit Regularization
25. Implications for Interpretability
26. Summary

Chapter 4: From Neurons to Layers: Computational Abstractions of Deep Learning

1. The Forward Pass as a Computational Graph
2. Composition and Data Flow
3. The Computational Graph View
4. Code: Visualizing a Computational Graph
5. Activation Functions and Nonlinearities
6. Common Activations
7. Why the Choice Matters
8. Code: Comparing Activation Functions
9. The Universal Approximation Theorem and Its Limits
10. What It Means
11. What It Does Not Mean
12. Representation Learning and Feature Hierarchies
13. The Idea of a Hierarchy
14. Code: Analyzing Representational Hierarchy
15. Backpropagation and the Flow of Information
16. The Chain Rule
17. Gradient Flow and Vanishing Gradients
18. Implications for Interpretability
19. Code: Visualizing Gradient Flow
20. Summary

Chapter 5: Transformer Architectures: Computational Graphs and Parameterized Computation

1. The Original Transformer Architecture
2. High-Level Structure
3. Token Embedding and Positional Encoding
4. The Transformer Layer
5. The Complete Forward Pass
6. Code: Transformer Forward Pass from Scratch
7. Attention as a Soft Selection Mechanism
8. Scaled Dot-Product Attention
9. Multi-Head Attention
10. Code: Understanding Attention Mechanics
11. Residual Connections and Layer Normalization
12. Residual Connections
13. Layer Normalization
14. Code: Comparing Normalization Variants
15. Modern Variants
16. Rotary Positional Embeddings (RoPE)
17. SwiGLU
18. Grouped Query Attention (GQA)
19. FlashAttention
20. Summary

Chapter 6: Embeddings, Positional Encoding and the Representation of Language

1. Tokenization and Vocabulary Construction
2. Why Subword Tokenization
3. Byte Pair Encoding
4. WordPiece and Unigram
5. Tokenization and Interpretability
6. Code: Exploring Tokenization
7. Token Embeddings as Learned Representations
8. Geometry of the Embedding Space
9. Probing Embeddings
10. The Embedding as Part of the Circuit
11. Absolute and Relative Positional Encodings
12. Sinusoidal Positional Encodings
13. Learned Positional Embeddings
14. Rotary Positional Embeddings (RoPE)
15. ALiBi (Attention with Linear Biases)
16. Code: Implementing Different Positional Encodings
17. Positional Encoding Mechanisms
18. Position in the Residual Stream
19. Position Through Attention Biases
20. Positional Awareness in Interpretability
21. Summary

Chapter 7: The Residual Stream as Information Highway

1. The Residual Stream: Definition and Dynamics
2. What Is the Residual Stream?
3. The Accumulator View
4. Code: Tracing the Residual Stream
5. Information Flow Analysis
6. What Information Is in the Residual Stream?
7. Measuring Information Content
8. Code: Probing the Residual Stream
9. Residual Stream Interference
10. Feature Competition
11. Cancellation and Cooperation
12. Code: Visualizing Interference
13. Experimental Studies of Residual Stream Structure
14. Empirical Observations
15. Key Findings from Literature
16. Summary

Chapter 8: Attention Mechanisms: From Self-Similarity to Inductive Patterns

1. Attention Head Specialization
2. Why Different Heads Do Different Things
3. Detecting Head Specialization
4. Code: Analyzing Attention Patterns
5. Pattern Recognition in Attention Maps
6. Head Types
7. Code: Detecting Head Types with TransformerLens
8. Induction Heads
9. What Are Induction Heads?
10. The Mechanism in Detail
11. Evidence for Induction Heads
12. Code: Detecting Induction Heads
13. Attention to Position and Syntax
14. Positional Attention
15. Syntactic Attention
16. Code: Analyzing Syntactic Attention
17. Attention Collapse and Redundancy
18. Not All Heads Are Useful
19. Measuring Head Importance
20. Summary

Chapter 9: Feed-Forward Networks and the Geometry of Feature Processing

1. MLP as Feature Transducers
2. The Standard MLP
3. The Expanded Dimension
4. SwiGLU in Modern Models
5. Code: Comparing MLP and SwiGLU
6. Feature Selectivity in Neurons
7. What Do Individual Neurons Respond To?
8. Activation Maximization
9. Code: Neuron Activation Analysis
10. Direct Logit Bias
11. MLP Contribution to Next-Token Prediction
12. Detecting Direct Logit Bias
13. Code: Analyzing Direct Logit Bias
14. MLP-Attention Interaction
15. How Attention and MLP Collaborate
16. Key-Value Memory
17. Summary

Chapter 10: Normalization, Initialization and the Stability of Deep Computation

1. Layer Normalization: Variants and Effects
2. Standard LayerNorm
3. Pre-Norm vs. Post-Norm
4. RMSNorm
5. Code: Comparing Normalization Variants
6. Parameter Initialization Strategies
7. Why Initialization Matters
8. Variance-Scaled Initialization
9. Transformer-Specific Initialization
10. Code: Analyzing Initialization Effects
11. Signal Propagation in Deep Networks
12. The Problem of Depth
13. Empirical Observations
14. Code: Measuring Residual Stream Growth
15. Normalization and Interpretability
16. How Normalization Affects Analysis Methods
17. LN-Free Models for Interpretability
18. Training Dynamics and Emergent Structure
19. How Training Shapes the Network
20. Summary

Chapter 11: The Nature of Learned Features: Superposition and Distributed Codes

1. The Feature-Dimension Mismatch
2. The Basic Problem
3. Superposition: More Features Than Dimensions
4. Code: Demonstrating Superposition
5. Superposition and Ghost Features
6. Ghost Features
7. Interference Patterns
8. Code: Visualizing Interference
9. Polysemanticity and Context Dependence
10. What Is Polysemanticity?
11. Context Dependence
12. Code: Detecting Polysemanticity
13. Sparse Versus Dense Representations
14. The Tradeoff
15. Activation Sparsity
16. Emergent Linear Structure in High Dimensions
17. The Linear Representation Hypothesis
18. Separability and Geometry
19. Summary

Chapter 12: Probing Representations and Testing Hypotheses

1. Probe Design and Training
2. The Basic Idea
3. Probe Types
4. Training Details
5. Code: Training Linear Probes
6. What Probes Can and Cannot Show
7. What Probes Show
8. What Probes Cannot Show
9. Code: Testing Probe Robustness
10. Temporal Probe Analysis
11. Tracking Representations Across Layers
12. Code: Layer-Wise Probing
13. Probe Robustness and Confounding
14. The Confounding Problem
15. Mitigation Strategies
16. Code: Controlled Probing
17. Probe Case Studies
18. Syntax Probing
19. Semantic Probing
20. Task-Specific Probing
21. Summary

Chapter 13: Feature Visualization and Activation Maximization

1. Gradient-Based Activation Maximization
2. The Basic Idea
3. Continuous Relaxation
4. Code: Activation Maximization in Embedding Space
5. Input Perturbation Methods
6. Changing Inputs to Target Activations
7. Token Substitution
8. Code: Token Substitution for Activation Maximization
9. Latent Space Exploration
10. Navigating Representation Space
11. Feature Visualization via Reconstruction
12. Feature Attribution Methods
13. Attributing Outputs to Features
14. Code: Simple Feature Attribution
15. Interpretation and Validation
16. How to Judge Visualizations
17. Avoiding Artifacts
18. Validation Methods
19. Summary

Chapter 14: Activation Patching and Logit Attribution

1. Mechanistic Interpretation Through Intervention
2. The Core Idea
3. Necessity and Sufficiency
4. Activation Patching: Method and Variants
5. The Basic Procedure
6. The Logit Difference Metric
7. Code: Basic Activation Patching with TransformerLens
8. Layer-Level and Head-Level Patching
9. Code: Head-Level Patching
10. Logit Attribution
11. Direct Logit Attribution
12. Code: Direct Logit Attribution
13. Patching Case Studies
14. Understanding Factual Knowledge
15. Understanding In-Context Learning
16. Summary

Chapter 15: Causal Tracing, Mediation and Circuit Discovery

1. Causal Tracing Methodology
2. Following Computation Through the Graph
3. Attribution Graphs
4. Code: Building a Causal Trace
5. Causal Mediation Analysis
6. Quantifying Information Flow
7. Application to Transformers
8. Code: Causal Mediation Analysis
9. Circuit Discovery Algorithms
10. Manual vs. Automated Discovery
11. Validation Criteria
12. Indirect Object Identification Circuit: A Worked Case Study
13. The IOI Task
14. The Discovered Circuit
15. How It Was Discovered
16. Code: IOI Circuit Analysis
17. Circuit Validation and Robustness
18. Testing Circuit Explanations
19. Backup Circuits
20. Summary

Chapter 16: Ablation Studies and Interventional Interpretability

1. Neuron and Head Ablation
2. Zeroing Out Components
3. Code: Attention Head Ablation
4. MLP Neuron Ablation
5. Gradual Versus Sudden Interventions
6. Sudden Ablation
7. Gradual Intervention
8. Code: Gradual Intervention
9. Ablation-Based Feature Importance
10. Ranking Components
11. Code: Component Ranking
12. Reconstruction and Substitution
13. Building Minimal Mechanisms
14. Substitution Experiments
15. Ablation Case Studies
16. What Ablations Have Revealed
17. Summary

Chapter 17: Sparse Autoencoders and Dictionary Learning for Feature Discovery

1. Sparse Autoencoders: Motivation and Formulation
2. The Problem with Neurons
3. The Autoencoder Architecture
4. Common Activation Functions
5. Code: SAE Architecture
6. Training Sparse Autoencoders
7. The Loss Function
8. Training Procedure
9. Key Hyperparameters
10. Code: Training an SAE
11. Dictionary Learning and Feature Quality
12. Evaluating Features
13. Feature Interpretation
14. Code: Feature Analysis
15. Transcoder Architectures
16. Alternatives to Standard SAEs
17. Choosing an Architecture
18. Scale and Generality
19. Scaling SAEs
20. Generality Across Models
21. Limitations
22. Summary

Chapter 18: Feature Engineering, Steering and Applied Interpretability

1. Steering Vectors
2. The Core Idea
3. Extracting Steering Vectors
4. Code: Difference of Means Steering
5. Feature Splitting and Absorption
6. How Features Evolve Across Layers
7. Code: Tracking Feature Directions
8. Model Diffing and Comparative Analysis
9. What Changes Between Models?
10. Applications
11. Interpretability for Safety and Alignment
12. Why Interpretability Matters for Safety
13. Current State
14. Challenges
15. The Future of Mechanistic Understanding
16. Open Problems
17. Emerging Directions
18. Summary

Conclusion: The Frontier of Understanding

References