About This Book
- What this book is
- What you will build
- How to use this book
- Acknowledgments
Chapter 1 — Introduction
- 1.1 What is a language model, really?
- 1.2 What "from scratch" means here
- 1.3 The road map
- 1.4 Tools and hardware
- 1.5 How the bilingual design shapes everything
- 1.6 Summary
Chapter 2 — The Dataset
- 2.1 A model is what it eats
- 2.2 Loading data with datasets
- 2.3 Exploring with pandas
- 2.4 Cleaning
- 2.5 Splitting: train / validation
Chapter 3 — Tokenization I: The Idea and the Algorithm
- 3.1 The bridge between text and numbers
- 3.2 Two naive schemes, and why they fail
- 3.3 Subword tokenization: the Goldilocks zone
- 3.4 BPE, from the beginning
- 3.5 Training BPE on the corpus
- 3.6 How to judge a tokenizer
Chapter 4 — Tokenization II: Encoding, French, and Production
- 4.1 Using what we trained
- 4.2 Special tokens
- 4.3 The French subtleties
- 4.4 Tokenizing and packing the corpus
Chapter 5 — PyTorch 101
- 5.1 Tensors
- 5.2 Autograd: the tape that trains everything
- 5.3 nn.Module and nn.Parameter
- 5.4 The training loop, once and forever
- 5.5 Devices: CPU, GPU, and moving things
- 5.6 Batching and DataLoader
- 5.7 Saving and loading: checkpoints
- 5.8 A complete tiny example: learning sine
Chapter 6 — Embeddings
- 6.1 From integer to geometry
- 6.2 Why learned? A parable about one-hot
- 6.3 Positions: the information attention can't see
- 6.4 The Entente embedding layer, complete
- 6.5 Seeing the space
Chapter 7 — Attention
- 7.1 The problem
- 7.2 Queries, keys, values
- 7.3 The score, the scale, and the softmax
- 7.4 The causal mask
- 7.5 Attention in seven lines
- 7.6 Multi-head: divide and conquer
- 7.7 The full layer, in PyTorch
- 7.8 What attention costs
Chapter 8 — The GPT Model
- 8.1 The residual stream
- 8.2 LayerNorm
- 8.3 The feed-forward network
- 8.4 The block, assembled
- 8.5 The full model
- 8.6 Config as contract
- 8.7 The sanity check that saves days
Chapter 9 — Pretraining
- 9.1 The loop, specialized
- 9.2 Reading the loss
- 9.3 AdamW and the learning-rate schedule
- 9.4 Clipping and accumulation
- 9.5 Watching it train: logging, sampling, Trackio
- 9.6 The two failure signatures
- 9.7 The actual commands
- 9.8 What pretraining teaches (and what it doesn't)
Chapter 10 — Text Generation
- 10.1 The generate loop
- 10.2 Greedy decoding
- 10.3 Temperature: flattening or sharpening
- 10.4 Top-k and top-p: cutting the tail
- 10.5 Reading generated text diagnostically
Chapter 11 — Finetuning with LoRA
- 11.1 From autocomplete to assistant
- 11.2 The expensive defaults
- 11.3 LoRA in one picture
- 11.4 The recipe
- 11.5 Evaluating a small chat model (honestly)
Chapter 12 — Evaluation and the Road Ahead
- 12.1 What you have, measured
- 12.2 The wider evaluation landscape
- 12.3 Directions from here
- 12.4 Closing
Appendix A — English–French ML Glossary
- Data & tokenization
- Model & training
- Generation & finetuning
Appendix B — Environment Setup
- B.1 Create the project and install
- B.2 Verify PyTorch and the GPU
- B.3 First run (smoke test)
- B.4 Tracking panel (Trackio)
- B.5 Free GPU options
- B.6 Project layout (what you build, chapter by chapter)
Appendix C — The entente Configuration Reference
- C.1 Model configuration (ModelConfig.entente())
- C.2 Pretraining configuration (TrainingConfig)
- C.3 LoRA finetuning configuration (run_finetune_lora.py)