Build an LLM Inference Engine in C++

Build an LLM Inference Engine in C++ — Through Challenges

You don't truly understand how large language models run

until you've built the engine yourself.

This book takes you from a blank C++ project to a complete,

working inference engine that loads a real Llama-family model

and generates text — one challenge at a time.

What you'll build:

- A strided tensor system with zero-copy views and arena allocation

- Math kernels: RMSNorm, SwiGLU, softmax, GEMM with SIMD

- A byte-level BPE tokenizer

- A full Transformer: RoPE, GQA, Flash Attention, KV Cache

- int8/int4 quantization with direct block multiplication

- GGUF model loading with mmap

- Sampling, streaming, speculative decoding, and continuous batching

- An optional CUDA capstone for the heaviest kernels

By Unit 14, the engine runs a real model on CPU. Every concept

earns its place right after you've built the thing it improves.

Who this is for: C++20 developers comfortable with algorithms

and memory layout who want to understand what actually happens

inside an LLM runtime — not by reading, but by building.