Evals

Somebody has shipped an AI feature, and somebody has asked you whether it is good.

If you came up through test automation, you already know the uncomfortable part: none of your instruments fit. You cannot diff the output. You cannot snapshot it. You cannot run it twice and expect the same thing — and the vocabulary you would normally reach for, flaky and regression and expected value, describes a defect rather than the thing in front of you, which is the product working as designed.

This book is the discipline that replaces those instruments.

WHAT'S INSIDE

Nine chapters, each one dense, none padded to reach a page count.

Why AI systems break classic testing — including why "just set the temperature to zero" fails, and why it would not help even if it worked.

The four-layer playbook — property families sorted by cost, from deterministic checks that run in milliseconds to judges that bill per token.

Golden datasets — the regression suite nobody hands you: anatomy of an entry, sizing and slicing, cold-start from the spec, the circularity trap, and why structured facts get exact matches back.

LLM-as-judge, and judging the judge — rubrics that measure quality instead of fluency, the four biases that corrupt your metrics, and how to calibrate a judge against human labels before you let it gate a release.

RAG evaluation — the triad, evaluating retrieval on its own with recall@k and MRR, why chunking is an eval parameter rather than a preprocessing detail, and a fourth metric the triad misses entirely: did the answer leave something out?

Adversarial and multi-turn evals — context retention, corrections that must supersede, escalation that carries its context, direct and indirect prompt injection, and the PII masking order that catches people out.

Running evals — per-intent gates, tiered economics, triggers beyond code, drift monitoring, and an honest maturity model most teams will not enjoy reading.

Agentic evals — scoring trajectories rather than answers, environments that must be rebuilt rather than reset, and the ways an agent games a verifier.

And a closing argument: the engineer who writes the eval is now, functionally, part of the training team.

HOW IT IS WRITTEN

Every concept arrives with its failure story — how it breaks in production, and how you catch it. The carrot cake that scored perfectly. The judge that gave a fabricated number nine out of ten. The escalation metric that stayed green while every handoff arrived empty. The concepts here are not difficult; recognizing them in the wild, at eleven at night, in someone else's codebase, is the skill, and that is learned from cases rather than definitions.

You will receive every future update free, and there will be updates — this field moves faster than print.