How Transformers Actually Work

From First Principles to Production Inference

Chapter 1: What Problem Are We Solving?

1. The Sequence Modeling Problem
2. What Came Before: RNNs, LSTMs, and Their Limits
3. Why Parallelism Matters
4. The Attention Insight
5. A Map of This Book

Chapter 2: Tensors, Shapes, and Matrix Multiplication

1. Tensors as n-Dimensional Arrays
2. Element-Wise Operations
3. Matrix Multiplication and Broadcasting
4. Batch Dimensions and the Last-Dimension Convention
5. A Running Example of Shape Tracking

Chapter 3: Neural Networks From First Principles

1. Linear Layers and Affine Transformations
2. Activation Functions: Why Non-Linearity Is Necessary
3. Loss Functions: Cross-Entropy and What It Measures
4. Backpropagation: The Chain Rule in Practice
5. Optimizers: Gradient Descent and Its Variants

Chapter 4: From Text to Numbers: Tokenization and Embeddings

1. Character-Level versus Token-Level Representations
2. Byte-Pair Encoding and Subword Tokenization
3. Building a Vocabulary and Handling Unknown Tokens
4. The Embedding Layer as a Lookup Table
5. Embeddings as Learned Vector Representations

Chapter 5: The Attention Mechanism

1. What Is Attention, Intuitively?
2. Query, Key, and Value: The Retrieval Analogy
3. Dot-Product Similarity and the Scaling Problem
4. Softmax as a Probability Distribution Over Positions
5. Computing Attention: A Step-by-Step Numerical Example

Chapter 6: Multi-Head Attention

1. Why One Head Is Not Enough
2. Projecting Into Subspaces
3. Concatenating and Mixing Head Outputs
4. The Tensor Algebra of Multi-Head Attention
5. What Different Heads Learn

Chapter 7: Positional Information

1. The Permutation Invariance Problem
2. Sinusoidal Positional Encodings: Derivation and Properties
3. Learned Positional Embeddings
4. Relative Position and Distance-Aware Attention
5. Masking: Why and When to Zero Out Positions

Chapter 8: Feed-Forward Networks

1. The Position-Wise Feed-Forward Network
2. Why We Need a Separate Transformation After Attention
3. Activation Functions: ReLU, GELU, and SwiGLU
4. Width, Depth, and Computational Trade-Offs
5. The Complete Transformer Block

Chapter 9: Implementing a Transformer in NumPy

1. Designing the Data Structures
2. The Embedding and Positional Encoding Layer
3. Implementing Scaled Dot-Product Attention From Scratch
4. Building Multi-Head Attention
5. The Complete Encoder Stack
6. Testing and Verification

Chapter 10: Building a Decoder and Encoder-Decoder Architecture

1. Causal Masking and Autoregressive Generation
2. The Decoder Self-Attention Layer
3. Cross-Attention: Attending to the Encoder
4. Output Projection and Logits
5. Generating Text One Token at a Time

Chapter 11: Automatic Differentiation and Backpropagation Through a Transformer

1. Gradients of Matrix Multiplication
2. Gradients Through Softmax and Attention
3. Gradients of the Feed-Forward Network
4. The Residual Connection and Gradient Flow
5. Verification Against Numerical Gradients

Chapter 12: Training a Transformer From Scratch With PyTorch

1. Why PyTorch: Autograd and Dynamic Graphs
2. Preparing a Tiny Training Dataset
3. The Complete Training Loop
4. Monitoring Loss and Generating Samples
5. Saving, Loading, and Resuming Checkpoints

Chapter 13: GPT-Style Decoder-Only Models

1. Why Decoder-Only Works for Language Generation
2. Architectural Choices in the GPT Family
3. Layer Normalization Placement and RMSNorm
4. RoPE: Rotary Positional Embeddings
5. SwiGLU and the Modern Feed-Forward Design

Chapter 14: BERT and Encoder-Only Models

1. Bidirectional Attention and Pretraining Objectives
2. Masked Language Modeling in Detail
3. Next Sentence Prediction
4. Fine-Tuning for Downstream Tasks
5. The Encoder-Only Limitations

Chapter 15: Encoder-Decoder and Instruction-Tuned Models

1. The T5 Architecture and Text-to-Text Pretraining
2. Prefix-LM and Hybrid Designs
3. Instruction Tuning and Alignment
4. Parameter-Efficient Fine-Tuning: LoRA and Friends
5. Architectural Trade-Offs in Practice

Chapter 16: Scaling Techniques and Mixture of Experts

1. The Scaling Laws: Data, Parameters, and Compute
2. Sparse Expert Routing
3. Mixture-of-Experts Training Dynamics
4. Load Balancing and Expert Capacity
5. MoE in Production Models

Chapter 17: Long-Context Techniques

1. The O(n²) Attention Bottleneck
2. Grouped-Query and Multi-Query Attention
3. Sliding Window Attention
4. Memory Compression and Token Pruning
5. Extending Context Through Data and Training

Chapter 18: Initialization, Optimization, and Scheduling

1. Initialization Strategies and Variance Control
2. AdamW and Why It Works for Transformers
3. Learning Rate Warmup and Cosine Decay
4. Gradient Clipping and Stability
5. Hyperparameter Recipes That Work

Chapter 19: Distributed Training: Parallelism Strategies

1. Data Parallelism and Gradient Synchronization
2. Model Parallelism: Tensor and Pipeline
3. ZeRO and Optimizer State Sharding
4. Communication Overhead and Topology
5. Real-World Training Runs

Chapter 20: Memory and Precision

1. The Memory Budget of a Transformer Training Run
2. Mixed Precision: FP16, BF16, and Loss Scaling
3. Gradient Accumulation for Effective Batching
4. Activation Checkpointing
5. Memory Profiling and Debugging

Chapter 21: Inference Optimization

1. Pre-Fill Versus Decode: Two Different Regimes
2. KV Caching: Why and How
3. Continuous Batching and Scheduling
4. Speculative Decoding
5. Speculative Decoding Correctness and Sampling

Chapter 22: Quantization and Efficient Formats

1. Why Quantization Works for Neural Networks
2. Post-Training Quantization: INT8 and Below
3. Quantization-Aware Training
4. Grouped Quantization and Outlier Handling
5. GGUF, GGML, and the Open-Source Ecosystem

Chapter 23: FlashAttention and GPU Hardware

1. GPU Memory Hierarchy and Tensor Cores
2. The Attention Kernel: Why It Is Slow
3. FlashAttention: IO-Aware Recomputation
4. FlashAttention-2 and Beyond
5. Measuring and Profiling GPU Utilization

Chapter 24: Serving Architectures

1. Throughput Versus Latency Trade-Offs
2. Model Serving Frameworks
3. Multi-GPU Inference and Expert Parallelism
4. Caching, Load Balancing, and Autoscaling
5. Cost Modeling and Efficiency

Chapter 25: What Transformers Can and Cannot Do

1. What Attention Can Represent
2. Transformers as Approximate Algorithm Implementers
3. Inductive Biases: What the Architecture Assumes
4. Positional Encoding Limits and Extrapolation
5. Failure Modes and Systematic Errors

Chapter 26: The History and Future of Attention

1. The Pre-Transformer Landscape: 2014-2017
2. “Attention Is All You Need”: Context and Impact
3. The Architecture Arms Race: 2018-Present
4. Post-Transformer Alternatives
5. Open Questions and Where to Look Next

Conclusion

1. The Transformer in One Page
2. What You Can Do Now
3. The Next Decade of Sequence Modeling

References