Deployment, Monitoring and Reliability

A deployed model is a model plus four other systems: a queue that decides how long requests

wait, a process lifecycle that decides what a restart costs, a configuration that decides what

the answer looks like, and an input distribution that changes while the model does not.

All four produce the same symptom — it used to work and now it does not — and the model absorbs

the blame because it is the most visible component and the least understood. A team that

responds by changing the model has spent its effort on the one thing that did not change.

This book measures the other four.

- **Waiting time is hyperbolic, not linear.** Raising utilisation from 0.50 to 0.95 — less than

doubling it — multiplied the mean wait by 19. Expressed against the service time: at half

capacity a request waits about as long as it is served, and at 0.95 it waits nineteen times

longer. The overwhelming majority of its life in the system is spent doing nothing.

- **A system under load and a system past capacity look identical at any instant.** Below

capacity the queue drains — at 0.90 utilisation the last quarter of requests waited 0.3x what

the first did. Past capacity it does not: at 1.20 utilisation they waited 4.1x. A dashboard

reporting nine seconds describes both. Only the direction distinguishes them, which inverts

the usual alerting rule: a high number that is falling is fine, and a moderate number that is

rising is an incident.

- **Irreproducibility is a choice.** Greedy decoding with a fixed seed agreed with itself 1.000

of the time. Changing temperature from 0.7 to 0.8 — a tenth of a point — dropped agreement to

0.164: five answers in six became different. "Same model, same prompt, different answer" is

almost always a configuration difference, and recording the seed rather than fixing it costs

a few bytes and makes any past request replayable.

- **Whether a good guardrail threshold exists depends on the signal, not the threshold.** With

well-separated populations one setting caught 93.5% of bad inputs while blocking 6.4% of good

ones. With a weaker signal, catching that same 93.5% required blocking 49.6% of legitimate

traffic. A team that keeps tuning a threshold and keeps being unhappy has diagnosed the wrong

component.

- **Drift detection is a sample-size question.** A shift of 0.1 standard deviations is invisible

in 100 samples and unmistakable in 1,000 — the same shift, the same test, a different amount

of evidence. Sensitivity and detection latency are one parameter seen from two ends.

- **Judging a release costs traffic, quadratically.** Detecting a five-point quality drop takes

685 requests per arm; a one-point drop takes 14,735. A null A/B result means nothing until the

smallest difference the test could have seen is computed.

- **A process is not ready when the file is open.** Loading a model was measured at roughly half

a millisecond per megabyte, of which only about 70% is reading — the rest is the first pass

that makes memory resident. A process can report itself loaded, pass its health check, and

charge a user for the pages nobody touched.

**On what this book measures and what it does not.** This is the most infrastructure-dependent

volume in the series, and it was written on a machine with one CPU core. That constraint was

faced before a single experiment was designed rather than after. Thirteen objectives were listed

from the published blueprints and then triaged: six are measured directly, two are simulated and

validated against closed-form results, and five are reasoned or cited. The ratio is printed on

the cover.

Where the book simulates, it says what that means: generating arrivals and service times from

known distributions and running the queue discipline exactly as a server would. Every simulated

result is checked against theory — the queue matches to three decimal places up to 0.95

utilisation, and where it stops matching, at 0.99, the book reports the discrepancy and explains

it rather than quietly presenting whichever number agrees. A simulation you have validated is

stronger evidence than a measurement you have not.

What you get:

- 9 chapters covering the four systems around a model, queueing and capacity, cold start and the

process lifecycle, configuration and reproducibility, drift, degradation dynamics, guardrails

and fallbacks and cost, releasing a change, and a final chapter of limits.

- 48 original practice questions, tagged by certification and by difficulty, where every option

is explained — not just why the right answer is right, but why each wrong answer is wrong,

because on these exams the distractors are where the teaching is.

- A reverse map from each certification to the chapters that serve it, in reading order.

- All the code and simulation scripts, so every table can be regenerated.

Written to the published objectives of NVIDIA NCP-GENL, NVIDIA NCA-GENL, AWS Certified

Generative AI Developer – Professional, Databricks Certified Generative AI Engineer, and NVIDIA

NCP Agentic AI. Objectives were checked in August 2026; confirm the current blueprint with the

certifying body before you sit.

You need to be able to read Python. Queueing, drift statistics and the arithmetic of thresholds

are built up from the point where they are needed, and nothing in this volume assumes you have

read any other book.

Every question is original, written from published exam objectives. Nothing is reproduced from,

or based on recollection of, any live examination.