Building Large Language Models from Scratch

Building Large Language Models from Scratch

A Practical Guide to Training Your Own Transformer-Based AI in Python

Introduction: Why Build from Scratch?

1. The Black Box Problem
2. What You Will Build
3. Prerequisites and How to Use This Book
4. A Note on Hardware Requirements

Chapter 1: Tokens, Vocabularies, and Tokenization

1. From Text to Numbers: The Tokenization Pipeline
2. Character-Level vs Word-Level vs Subword Tokenization
3. Building a Byte-Pair Encoding (BPE) Tokenizer
4. Vocabulary Size, Special Tokens, and Edge Cases
5. Exercise: Tokenize the Shakespeare Corpus

Chapter 2: Embeddings: Turning Tokens into Vectors

1. The Embedding Layer as a Lookup Table
2. Learning vs Pre-trained Embeddings
3. Vector Space Geometry and Similarity
4. Implementing Embedding Layers in PyTorch
5. Exercise: Visualize an Embedding Space

Chapter 3: The Attention Mechanism

1. What Is Attention and Why It Matters
2. Scaled Dot-Product Attention from First Principles
3. Multi-Head Attention: Parallelizing Understanding
4. Causal Masking for Decoder-Only Models
5. Exercise: Trace Attention Through a Sentence

Chapter 4: Positional Encoding and Sequence Structure

1. The Permutation Invariance Problem
2. Sinusoidal Absolute Positional Encodings
3. Learned Position Embeddings
4. Rotary Positional Embeddings (RoPE)
5. Exercise: Implement Three Position Encoding Schemes

Chapter 5: Building the Decoder-Only Transformer Architecture

1. The Decoder-Only Design Decision
2. Feed-Forward Networks and MLP Blocks
3. Residual Connections and Layer Normalization
4. Assembling the Full Transformer Block
5. The Complete Decoder-Only Model
6. Exercise: Build a 3-Layer Decoder from Scratch

Chapter 6: Data Preparation for Language Model Training

1. The Data Landscape: Where Does Training Data Come From?
2. Cleaning and Filtering Pipeline Design
3. Deduplication Strategies
4. Dataset Mixing and Domain Balancing
5. Synthetic Data Generation Strategies
6. Exercise: Build a Mini C4 Dataset

Chapter 7: The Training Loop: Loss, Optimizers, and Gradient Flow

1. Cross-Entropy Loss and Next-Token Prediction
2. The AdamW Optimizer and Why It Works
3. Learning Rate Schedules: Warmup, Cosine Decay, and Beyond
4. Gradient Clipping and Training Stability
5. The Complete Training Loop
6. Exercise: Train on a Tiny Dataset and Monitor Loss Curves

Chapter 8: Memory-Efficient Training Patterns

1. The Memory Wall: Why Models Don’t Fit in GPU RAM
2. Mixed-Precision Training with BF16/FP16
3. Gradient Accumulation for Effective Batch Sizes
4. Activation Checkpointing and Recomputation
5. Exercise: Train a 10x Larger Model on the Same Hardware

Chapter 9: Distributed Training and Parallelism Strategies

1. Data Parallelism and Distributed Data Parallel (DDP)
2. Tensor Parallelism for Massive Models
3. Pipeline Parallelism: Splitting the Forward Pass
4. Fully Sharded Data Parallel (FSDP)
5. Exercise: Multi-GPU Training Setup

Chapter 10: Checkpointing, Experiment Tracking, and Reproducibility

1. Checkpointing Strategies and Recovery
2. Experiment Tracking: Metrics, Configs, and Artifacts
3. Reproducibility: Seeds, Determinism, and Hardware Variability
4. Logging Design for Long Training Runs
5. Exercise: Set Up a Production-Style Training Dashboard

Chapter 11: Fine-Tuning: LoRA, QLoRA, and Instruction Tuning

1. The Fine-Tuning Landscape: Full vs Parameter-Efficient
2. Low-Rank Adaptation (LoRA) from First Principles
3. Quantized LoRA (QLoRA) for Memory-Constrained Fine-Tuning
4. Instruction Tuning Dataset Design
5. Exercise: Fine-Tune a Model on a Custom Task

Chapter 12: Alignment: RLHF and Beyond

1. Why Raw Models Need Alignment
2. The RLHF Pipeline: Reward Models and PPO
3. Direct Preference Optimization (DPO) as a Simpler Alternative
4. Constitutional AI and Rule-Based Alignment
5. Exercise: Build a Simple Preference Dataset

Chapter 13: Evaluation: Metrics, Benchmarks, and Red Teaming

1. Perplexity as a Training Metric vs Real-World Performance
2. Standard Benchmark Suites (MMLU, GSM8K, HumanEval)
3. Qualitative Evaluation and LLM-as-Judge
4. Safety Testing and Red Teaming Methodologies
5. Exercise: Build an Evaluation Harness

Chapter 14: Deployment: Inference, Quantization, and Serving

1. Inference Optimization: KV Cache and Speculative Decoding
2. Quantization Strategies: FP16 -> INT8 -> INT4
3. Building a Serving API with FastAPI
4. Containerization and Production Deployment Patterns
5. Exercise: Deploy Your Model Behind a Live API

Capstone Project: From Raw Data to Live Inference

1. Project Setup and Architecture Overview
2. Data Preparation: Curating a 100M-Token Dataset
3. Training Run: Configuration, Execution, and Monitoring
4. Evaluation: Benchmarking Against Baselines
5. Deployment: Containerized API with Health Checks
6. Post-Mortem: What Went Well and What Would Be Different

Conclusion: The Road Ahead

1. What You Have Accomplished
2. Where to Go From Here: Scaling Up
3. The Future of Language Models
4. A Final Word on Building vs Using

References