Optimisation and Acceleration

Inference optimisation is the part of this field where the techniques are most numerous and the

reasoning behind them least examined. Each one is real, and they are usually applied in the

order someone heard of them — producing a serving system with several optimisations, no

instrumentation, and no way to say which one is helping.

This book measures the underlying ratio first and then measures every technique against it. The

reference machine was clocked at 22.54 GB/s of memory bandwidth and 116.58 GFLOP/s of

arithmetic, giving a ridge point of 5.17 operations per byte. Generating one token performs

0.22 — twenty-three times below it. That single fact explains why decoding is slow, why

batching works, why the cache dominates long contexts, and why a second model can run almost

for free.

Several results are not what the technique's reputation suggests:

- **Unstructured pruning saves nothing.** Zeroing 60% of the weights cost 5% of perplexity and

not one byte of memory, because a dense matrix full of zeros is the same size as one full of

numbers. A sparse format costs positions as well as values and does not pay until roughly two

thirds sparsity.

- **Structured pruning destroyed the model — and that is not the verdict.** Perplexity went from

146.02 to 2569.46, and fine-tuning afterwards brought a model with a *quarter* of its

feed-forward channels back to 162.68, within 11% of the original. Pruning is

prune-then-recover, and the intermediate number should never be evaluated.

- **Batch 2 is slower than batch 1.** A real dispatch discontinuity, kept and explained rather

than smoothed away — the throughput curve is not smooth, and a batch size chosen by reasoning

rather than measurement can land in a hole.

- **An unrelated draft model accepts nothing.** Speculative decoding was verified as producing

output *identical* to greedy decoding, and then measured against a control: an ordinary model

of similar size accepted 0.000 of its proposals at every draft length. "A small fast model" is

an incomplete description of what a draft must be.

- **Tensor parallelism exchanges almost no data** — between 0.002% and 0.036% of the weight

bytes each device reads — and is still confined to one machine, because the all-reduce is a

synchronisation point hit once per block per token.

- **The largest saving is work not done.** A request sharing a 1,024-token prompt prefix with an

earlier one was 72.6x faster, superlinearly, because prefill attention is quadratic.

- **Memory and compute can be pulled apart.** A mixture-of-experts model with 128 experts and

one active per token holds 70.2x the parameters of its dense baseline and performs exactly the

same arithmetic — so "smaller model" and "less work per token" stop being the same statement.

And the ones that confirm their reputation come with the numbers: 8-bit quantisation costs 0.0%

of perplexity at 4x compression; granularity governs the cliff below that, cutting 4-bit damage

from 8.6% to 3.2%; batch 32 gives 3.9x the throughput for 8.2x the per-step latency; and at

131,072 tokens the KV cache is twenty-three times the size of the model.

**On quantisation, and a companion book.** Quantisation has an entire book of its own —

*LLM Quantization: From the Bits Up*, by the same author, 104 pages building the subject from

the representation of a single number upward. Chapter 2 of this volume covers quantisation to

the full depth these certifications require, measured, and refers the derivations, the named

algorithms and bit-level packing onward for readers who want them. You do not need that book to

pass the exams or to use this one; Chapter 2 stands alone.

What you get:

- 12 chapters covering where inference time goes, quantisation, pruning, distillation, the KV

cache and its remedies, speculative decoding, batching and continuous batching, throughput

against latency, multi-device parallelism, prefix caching and chunked prefill,

mixture-of-experts, and a final chapter of limits.

- 66 original practice questions, tagged by certification and by difficulty, where every option

is explained — not just why the right answer is right, but why each wrong answer is wrong,

because on these exams the distractors are where the teaching is.

- A reverse map from each certification to the chapters that serve it, in reading order.

- All the code and measurement scripts, so every table can be regenerated.

This volume covers the two heaviest domains on the NVIDIA NCP-GENL blueprint — Model

Optimization and GPU Acceleration — which together carry 31% of that exam.

Written to the published objectives of NVIDIA NCP-GENL, NVIDIA NCA-GENL, AWS Certified

Generative AI Developer – Professional, Databricks Certified Generative AI Engineer, and NVIDIA

NCP Agentic AI. Objectives were checked in August 2026; confirm the current blueprint with the

certifying body before you sit.

You need to be able to read Python. Attention, the KV cache and the arithmetic of memory traffic

are built up from the point where they are needed, and nothing in this volume assumes you have

read any other book.

Every question is original, written from published exam objectives. Nothing is reproduced from,

or based on recollection of, any live examination.