AI Engineering Visualized, Volume 4: Production AI

Asha's mug set arrived chipped, and she wants a refund. The worker calls the payment provider, and the process dies before it writes anything down. Was she paid?

That is the question Chapter 1 answers. Through three volumes the Northwind Goods assistant learned to answer, to act safely and to answer from real knowledge. It did all of that in one process on one laptop, for one customer at a time. This book runs it for real users: many tickets at once, on machines that fail, with releases that can go wrong and costs that add up.

For people who have a working assistant or agent and now have to operate it. Volume 1 built the Northwind Goods support assistant. Volume 2 put a harness around it so it could act. Volume 3 gave it real knowledge. This volume takes it to production.

One assistant, carried forward. The same support assistant for Northwind Goods, the fictional online store, now run as a service. Chapter by chapter you build what surrounds the model in production, and every chapter adds one part to it:

• Boundaries and targets (Chapters 1-3): an API, a queue and leased workers with identity, secrets and state at named boundaries; SLOs counted as events, with error budgets and a written policy; traces, metrics you can trust and logs with nothing private in them.
• Knowing whether it works (Chapters 4-6): live traffic scored in the shadow and watched for drift, feedback and experiments, and releases you can version, test and roll back.
• Fast and affordable (Chapters 7-12): the critical path, streaming and backpressure, tokens and caching, model routing and fallbacks, serving choices, and when to fine-tune, distil or change models.
• Safe and recoverable (Chapters 13-16): threat modelling and red-teaming, privacy and the data lifecycle, incident response, and a capstone that operates the complete system.

Visual first. This edition has 36 named mental models, such as Request Path, Job Path, The SLO Card, Canary and Rollback, The Critical Path Waterfall, The Attack Surface Map or The Weekly Improvement Loop. A mental model here is a named picture you can reuse when you design or debug. Each one has a hand-drawn doodle, a plain definition, a walkthrough, and why it matters; most also break the running example and fix it.

Real code, no API key. The examples are short Python excerpts from the free companion repository (MIT licence), in its vol4/ folder at tag vol4-ed1. The book builds plain parts with the Python standard library so that every step is code you can read. Further Reading points to the real tools and standards. By default the code calls a stubbed model, with no API key, no network and no cost. Traffic, durations and failures come from a seeded simulator, and the book labels them simulated. You can point later chapters at a real local model with Ollama if you like. A small evidence pack of local-model runs, with model digest, settings, hardware and sample size, is in the Experiment Ledger appendix. Everything else is labelled simulated or unmeasured.

Chapters 1, 6 and 15 are in the free sample.

What you'll learn

• Put the assistant behind an API and a queue of leased workers, with identity, secrets and state at named boundaries
• Write SLOs counted as events, with error budgets and a written policy
• Trace every request, trust metrics only when the spans are complete, and log nothing private
• Score live traffic in the shadow, off the reply path, and watch for drift
• Turn feedback into tests and experiments, and ship a change only when the difference is real
• Version every change in a release you can test and roll back
• Shorten the critical path, stream under a contract, and apply backpressure
• Price cost per resolved ticket, and cache with a stable prompt prefix
• Route each ticket type to a model with tested fallbacks
• Choose managed or self-hosted serving, and decide when to fine-tune, distil or change models
• Threat-model the attack surface, check trust boundaries in code, and red-team the system
• Map personal data through its lifecycle, and keep an audit trail
• Contain, recover and verify after an incident
• Operate the complete system for a week and report it honestly

Who it's for

People who have a working assistant or agent and now have to operate it. You should be comfortable reading Python. You do not need a machine learning or operations background. Volume 1, or the same ground (the whole-system diagram, tokens, evals, guardrails and privacy, and the timeouts, retries and latency budget of its last chapter), comes first. Volumes 2 and 3 help lightly: the code starts from Volume 3's final code, which contains Volume 2's agent harness, and each piece it reuses is recapped in one line. You need Python 3.10 or newer and git.

Who it's not for

If you want a tour of production platforms, this isn't that book. It will not hide the system in a platform. It builds plain parts with the Python standard library so that every step is code you can read. Further Reading points to the real tools and standards.

Free sample

Front matter plus Chapters 1, 6 and 15.

Complete

All 16 chapters and the appendices are written and illustrated in this edition (292 pages in the screen PDF). Buy once and get every future revision free.

Table of contents

Boundaries and targets

1. Production Architecture: APIs, Workers, State and Boundaries

2. SLOs and Error Budgets

3. Observability: Traces, Metrics and Logs

Knowing whether it works

4. Online Evaluation: Shadow Scoring and Drift

5. Feedback and Experiments

6. Release Engineering: Versioning, CI and Rollbacks

Fast and affordable

7. Latency Engineering: The Critical Path

8. Streaming, Cancellation and Backpressure

9. Cost Engineering: Tokens and Caching

10. Model Routing, Fallbacks and Provider Resilience

11. Inference and Serving: Managed vs Self-Hosted

12. When to Fine-Tune, Distill or Change Models

Safe and recoverable

13. Security Engineering: Threat Modeling and Red-Teaming

14. Privacy, Governance and Data Lifecycle

15. Incident Response: Contain, Recover, Verify

16. Capstone: Operating the Complete System

Appendix: Capstone, One Week Through Every Box

Plus: Further Reading, a Joined Drill, Transfer Exercises, a Decision Guide, an Experiment Ledger, Audit Log Integrity, Quick Reference, a Solutions Index, Quiz Answers, a Glossary and a Mental-Model Index.