Building Production Voice AI Agents

Two voice agents answer the same customer question. One replies in about a second and a half. The other leaves six seconds of silence, long enough to make a real caller wonder if the line dropped. This book starts by measuring exactly where that silence goes, and does not stop measuring for the next thirty-four chapters. Building Production Voice AI Agents is a hands-on, evidence-first guide to building real-time voice agents that actually work on a real telephone network, not just in a browser demo. Every number in it comes from a real recorded run, shipped in the companion repository so you can check it yourself without an API key. Every bug it shows was found by placing a real call, not invented for the page. What you'll build: - A cascaded and a realtime voice agent, measured and compared on the same real question, not assumed. - A real telephone leg: SIP, RTP, DTMF, and a self-hosted LiveKit media server, no managed platform hiding what's actually happening underneath. - Tools with a real ledger, idempotency keys, and read-back confirmation, because a booking and a refund should never be treated as equally safe. - Multi-agent handoffs that survive being spoken instead of typed, plus the routing loop and the capability leak that happen when they don't. - A real evaluation harness: scenarios, invariants, a leaderboard that refuses to name a winner when the evidence doesn't support one, and a production audit that reruns itself against a fresh checkout. - A data-driven agent registry, a Studio UI to version and deploy it, canary releases with real rollback, and a multi-tenant boundary tested with 18 real cross-organization attacks, all refused. - A real production deployment: a self-hosted box, two real dispatch bugs found only because a phone was still ringing, and a live call bridged from Germany to Kenya to prove it. Who this is for: you can read Python, use a terminal, and you've probably built or used a text-based AI agent before. You don't need to know anything about digital audio, the telephone network, or real-time systems beyond using one; every idea in this book is built from what the chapter before it already established. Why this book is different: most voice-AI content teaches you to call a managed API and trust what it hides. This book teaches the self-hosted path deliberately: LiveKit, real SIP trunks, a real cloud box, because understanding what a managed platform hides is what lets you judge whether to trust it. When something breaks, real accounts of the discovery are used, not invented ones, and when a run doesn't fully explain itself, the book says so instead of smoothing it into a success story the numbers don't support. Thirty-five chapters, three appendices, and a Git tag for every one of them. Read it start to end, or open the companion repository and follow along chapter by chapter.