Evaluating Gen AI Applications: A Practitioner’s Playbook for Testing What AI Actually Gets Right

<p><strong>Second Edition — July 2026</strong></p>

<p>Gen AI applications rarely fail with a clean error message. They fail while producing fluent, persuasive and apparently useful answers. <em>Evaluating Gen AI Applications</em> gives engineering, testing, product, risk and governance teams a practical operating model for finding those failures before users, auditors or regulators do.</p>

<p>Using the continuing Meridian case, the book shows how to turn business requirements, failure modes and production incidents into versioned evaluation datasets, measurable checks, calibrated human review, regression suites, release gates, monitoring signals and audit-ready evidence. It covers prompts, RAG systems, agents, multimodal applications, code and reasoning, fine-tuning, model migrations, red teaming, retrieval authority, workflow economics and production governance.</p>

<p>The Second Edition strengthens uncertainty treatment, paired comparisons, retrieval denominators, the correct pass@k estimator, protected-behaviour gates and evidence-linked release decisions. It also adds deeper treatment of MCP and agent-protocol security, memory failures, C2PA provenance, current OWASP and MITRE ATLAS guidance, and seven interactive browser-based companion applications that let readers practise the book’s evaluation decisions using realistic Meridian scenarios.</p>

<p>This is not a prompt collection or a catalogue of benchmark scores. It is a practitioner’s guide to deciding what must be tested, which evidence is credible, when a release should be stopped and how evaluation should continue after deployment.</p>