LLM evaluation for developers

The demo went well. What nobody can say yet is whether the feature works for the people using it, or whether it will still work after the next prompt tweak, model swap or change to retrieval.

The book starts from what you already have, unit tests and CI, and shows where they stop helping once a model sits behind the code. From there it works through evaluation on one system. You read traces and name the failures that actually matter, then build a golden set from them. You decide which checks belong in code, which need an LLM judge and which need a person. You design a judge and calibrate it against human labels. A chapter on statistics shows how to tell a real improvement from noise when the set is small. Retrieval is evaluated apart from generation, tool calls and agents get their own chapter, and it all ends up as regression gates in CI, with sampled production traffic and a look at what the whole thing costs.

Every chapter ends with a decision you could make on Monday. The final chapter takes the evaluation questions AI engineering interviewers ask and answers them starting from the trade-off.

Examples are in Python, each one run before publication, with a note wherever TypeScript does the job differently. Claims about papers, libraries and techniques are checked against the primary source. It's about evaluating products built on language models, not training them. You need to be able to program. You don't need machine learning or statistics.