Building Large Language Models from Scratch

A Practical Guide to Training Your Own Transformer-Based AI in Python

Introduction: Why Build from Scratch?

1. The Black Box Problem
2. What You Will Build
3. Prerequisites and How to Use This Book
4. A Note on Hardware Requirements

Chapter 1: Tokens, Vocabularies, and Tokenization

1. From Text to Numbers: The Tokenization Pipeline
2. Character-Level vs Word-Level vs Subword Tokenization
3. Building a Byte-Pair Encoding (BPE) Tokenizer
4. Vocabulary Size, Special Tokens, and Edge Cases
5. Walkthrough: Training and Analyzing a BPE Tokenizer on Shakespeare

Chapter 2: Embeddings: Turning Tokens into Vectors

1. The Embedding Layer as a Lookup Table
2. Learning vs Pre-trained Embeddings
3. Vector Space Geometry and Similarity
4. Implementing Embedding Layers in PyTorch
5. Walkthrough: Visualizing an Embedding Space

Chapter 3: The Attention Mechanism

1. What Is Attention and Why It Matters
2. Scaled Dot-Product Attention from First Principles
3. Multi-Head Attention: Parallelizing Understanding
4. Causal Masking for Decoder-Only Models
5. Walkthrough: Tracing Attention Through a Sentence

Chapter 4: Positional Encoding and Sequence Structure

1. The Permutation Invariance Problem
2. Sinusoidal Absolute Positional Encodings
3. Learned Position Embeddings
4. Rotary Positional Embeddings (RoPE)
5. Exercise: Implement Three Position Encoding Schemes

Chapter 5: Building the Decoder-Only Transformer Architecture

1. The Decoder-Only Design Decision
2. Feed-Forward Networks and MLP Blocks
3. Residual Connections and Layer Normalization
4. Assembling the Full Transformer Block
5. The Complete Decoder-Only Model
6. Exercise: Build a 3-Layer Decoder from Scratch

Chapter 6: Mixture of Experts Architectures

1. Why Mixture of Experts?
2. How MoE Layers Work
3. Switch Transformers and Single-Expert Routing
4. Implementing an MoE Transformer Block
5. Training Considerations for MoE Models
6. Notable MoE Models
7. Exercise: Train a Tiny MoE Model

Chapter 7: Data Preparation for Language Model Training

1. The Data Landscape: Where Does Training Data Come From?
2. Cleaning and Filtering Pipeline Design
3. Deduplication Strategies
4. Dataset Mixing and Domain Balancing
5. Synthetic Data Generation Strategies
6. Exercise: Build a Mini C4 Dataset

Chapter 8: The Training Loop: Loss, Optimizers, and Gradient Flow

1. Cross-Entropy Loss and Next-Token Prediction
2. The AdamW Optimizer and Why It Works
3. Learning Rate Schedules: Warmup, Cosine Decay, and Beyond
4. Gradient Clipping and Training Stability
5. The Complete Training Loop
6. Exercise: Train on a Tiny Dataset and Monitor Loss Curves

Chapter 9: Scaling Laws and Compute-Optimal Training

1. The Empirical Laws of Language Model Scaling
2. Chinchilla and Compute-Optimal Training
3. Applying Scaling Laws to Your Training Run
4. Understanding the Limits of Scaling
5. Exercise: Analyze Scaling on Your Hardware

Chapter 10: Memory-Efficient Training Patterns

1. The Memory Wall: Why Models Don’t Fit in GPU RAM
2. Mixed-Precision Training with BF16/FP16
3. Gradient Accumulation for Effective Batch Sizes
4. Activation Checkpointing and Recomputation
5. Exercise: Train a 10x Larger Model on the Same Hardware

Chapter 11: Distributed Training and Parallelism Strategies

1. Data Parallelism and Distributed Data Parallel (DDP)
2. Tensor Parallelism for Massive Models
3. Pipeline Parallelism: Splitting the Forward Pass
4. Fully Sharded Data Parallel (FSDP)
5. Exercise: Multi-GPU Training Setup

Chapter 12: Checkpointing, Experiment Tracking, and Reproducibility

1. Checkpointing Strategies and Recovery
2. Experiment Tracking: Metrics, Configs, and Artifacts
3. Reproducibility: Seeds, Determinism, and Hardware Variability
4. Logging Design for Long Training Runs
5. Exercise: Set Up a Production-Style Training Dashboard

Chapter 13: Fine-Tuning: LoRA, QLoRA, and Instruction Tuning

1. The Fine-Tuning Landscape: Full vs Parameter-Efficient
2. Low-Rank Adaptation (LoRA) from First Principles
3. Quantized LoRA (QLoRA) for Memory-Constrained Fine-Tuning
4. Instruction Tuning Dataset Design
5. Exercise: Fine-Tune a Model on a Custom Task

Chapter 14: Alignment: RLHF and Beyond

1. Why Raw Models Need Alignment
2. The RLHF Pipeline: Reward Models and PPO
3. Direct Preference Optimization (DPO) as a Simpler Alternative
4. Constitutional AI and Rule-Based Alignment
5. Walkthrough: Training a Model with Direct Preference Optimization

Chapter 15: Evaluation: Metrics, Benchmarks, and Red Teaming

1. Perplexity as a Training Metric vs Real-World Performance
2. Standard Benchmark Suites (MMLU, GSM8K, HumanEval)
3. Qualitative Evaluation and LLM-as-Judge
4. Safety Testing and Red Teaming Methodologies
5. Exercise: Build an Evaluation Harness

Chapter 16: Deployment: Inference, Quantization, and Serving

1. Inference Optimization: KV Cache and Speculative Decoding
2. Quantization Strategies: FP16 -> INT8 -> INT4
3. Building a Serving API with FastAPI
4. Containerization and Production Deployment Patterns
5. Exercise: Deploy Your Model Behind a Live API

Chapter 17: Advanced Inference: Speculative Decoding and Beyond

1. The Inference Bottleneck
2. How Speculative Decoding Works
3. Practical Speedups and Real-World Performance
4. N-Gram Speculation
5. Exercise: Implement and Benchmark Speculative Decoding

Chapter 18: Context Window Extension and Long-Sequence Techniques

1. The Context Window Problem
2. Positional Encoding Interpolation
3. YaRN: Yet Another RoPE Extension
4. Attention Sparsity for Long Sequences
5. Practical Recommendations for Long Context

Chapter 19: Retrieval-Augmented Generation (RAG)

1. Why RAG? The Knowledge Problem
2. The RAG Pipeline
3. Building a Simple RAG System
4. RAG vs Fine-Tuning

Chapter 20: Model Compression Beyond Quantization

1. The Compression Landscape
2. Pruning: Removing Unnecessary Weights
3. Knowledge Distillation: Teaching Smaller Models
4. Combining Compression Techniques

Capstone Project: From Raw Data to Live Inference

1. Project Setup and Architecture Overview
2. Data Preparation: Curating a 100M-Token Dataset
3. Training Run: Configuration, Execution, and Monitoring
4. Evaluation: Benchmarking Against Baselines
5. Deployment: Containerized API with Health Checks
6. Post-Mortem: What Went Well and What Would Be Different

Conclusion: The Road Ahead

1. What You Have Accomplished
2. Where to Go From Here: Scaling Up
3. The State of Language Models in 2025
4. A Final Word on Building vs Using

References