A-to-Z Transformer Walkthrough: “The capital of France is ___?”

Transformers power many of today’s most capable AI systems—but understanding what actually happens inside a Transformer can be surprisingly difficult. Diagrams and equations often explain the individual components, yet it can still be hard to see how all those pieces work together to turn a sequence of words into a next-token prediction.

A-to-Z Transformer Walkthrough takes a different approach.

Using the simple prompt “The capital of France is ___?”, this book follows a deliberately small, transparent decoder-only Transformer from beginning to end. Rather than hiding the calculations behind high-level deep-learning libraries, it exposes the numbers and mathematics at each major stage.

You will follow the journey through:

• tokenization and vocabulary construction
• token embeddings
• sinusoidal positional encoding
• Query, Key, and Value (Q/K/V) projections
• scaled dot-product attention
• causal masking
• multi-head self-attention
• residual connections and LayerNorm
• feed-forward networks (FFNs)
• multiple Transformer layers
• the final hidden representation
• the language-model head
• logits and softmax probabilities
• next-token prediction
• backpropagation and training with Adam

The walkthrough begins with randomly initialized weights, allowing you to see why an untrained Transformer does not magically know that the answer is Paris. The same miniature model is then actually trained, showing how gradient-based learning changes its parameters until Paris becomes the overwhelmingly likely next-token prediction.

The book also discusses what this tiny experiment does not prove, including memorization, shortcut learning, and the limitations of training on very small datasets.

Complete reproducible Python code is included so that you can run the model yourself and inspect the calculations.

This is not intended to be a production implementation of GPT. It is an educational, simplified GPT-style decoder-only Transformer designed for one purpose:

to make the machinery inside a Transformer tangible, traceable, and easier to understand—one step and one number at a time.