Chapter 01 - Your PoC Is Not in Production

The preface made a claim. This chapter defends it.

The claim is that agents are software. The engineering discipline you already have for APIs - versioned schemas, contract tests, SLOs, canaries, rollback procedures, on-call rotations - is the discipline that is missing from how teams ship LLMs today. The claim is easy to nod at and harder to act on. The gap between nodding and acting is the gap this book exists to close.

This chapter explains three failure modes, drawn from systems already shipped and quietly degraded, each one showing what the gap looks like in the room. The first is a retrieval failure that produces wrong answers without anyone noticing. The second is a prompt change that breaks an SLA three days later. The third is a cost blowup that nobody saw until the invoice arrived.

The examples use different surfaces - a financial-document chatbot, a customer-facing classifier, an internal support agent. From chapter 2 onward, you apply the same discipline to a fourth: an open-source maintainer agent triaging GitHub issues in triage-lab. Different corpus. Same failure mechanics.

These are not the only failure modes that matter. They are the three that map most cleanly onto the discipline you already have, and the discipline this book asks you to bring. After all three, the pattern is impossible to miss. After the pattern, the rest of the book stops feeling like a sequence of opinions and starts feeling like a checklist.

If you have shipped an LLM proof-of-concept and are now being asked to ship something real, at least one of these failure modes is already happening to your team. You may not know yet.

A RAG agent over a corporate financial document. It has been live for two months. Stakeholders are happy. The chatbot returns answers that sound authoritative.

A user asks a reasonable question - “How many employees does the company have”? The answer is in the document, on a single page in the Human Capital Resources section. The agent retrieves three chunks from the index, none of which contain the figure. It refuses to answer: “I could not find this in the provided document”.

The refusal looks correct. It is not. The information is in the document; it is on page 16. The retriever ranked adjacent pages higher because naive fixed-size chunking split the relevant paragraph across the page 15–16 boundary. The model received chunks about compensation programs and operations centers. It correctly declined to invent a number from the wrong context. From the user’s perspective, the system simply does not know.

Nothing in the production system noticed this. No exception was thrown. No alert fired. The agent responded in 1.9 seconds with a fluent, well-formatted refusal. The user shrugged, asked something else, and within a week stopped using the system. They did not complain. They did not file a ticket. Their trust eroded silently.

This is a complete failure of the production system. It is also a failure that is invisible to every metric the team is currently watching. Latency is fine. Error rate is fine. Token throughput is fine. The only signal that would have caught this lives in a field that the team does not log: the retrieval score of the top-ranked chunk. In this case, that score was 0.176. The threshold below which a retrieval is suspect is around 0.75. The system answered with full confidence on a request whose retrieval was four times below the floor.

The fix is not complicated. It is one structured log field per request, plus a moving average over a rolling window, plus an alert when the moving average of low-confidence retrievals exceeds 10%. It is also a small golden dataset of (query, expected page) pairs that runs in CI, asserting that a chunking change does not drop the retrieval hit rate below 75%. None of this is novel. It is the same discipline that already applies to your API: log the field you care about, set a threshold, alert when the threshold is crossed, and write the test that prevents the regression in the first place.

What was missing was not a tool. It was the engineering posture that says “I cannot ship this without measuring it”. Without that posture, the team measured everything except the thing that decided whether the system was useful. With it, the failure mode is impossible to deploy in the first place.

On a document corpus, the miss was a page boundary. On an issue corpus, the same miss is a chunk boundary - title in one chunk, the duplicate issue ID in a comment thread the retriever never indexed - or a low similarity score on genuinely related closed issues while the model still answers with confidence. The maintainer agent in chapter 2 has no retrieval layer yet; when it mislabels a duplicate, that is often why.

Chapter 5 builds the retrieval layer of the maintainer agent and includes the golden dataset and the CI gate. Chapter 8 makes the retrieval-quality assertion part of the eval suite. Chapter 9 makes the retrieval score a first-class observability metric in production. Three chapters; one failure mode closed.

A team has a customer-facing classification agent running in production. It has been stable for three months. On a Tuesday, an engineer opens a pull request titled “small prompt clarity improvement”. The diff is fifteen lines of system prompt - clarifying a category definition, adjusting a few examples in the few-shot block. The eval suite passes. CI is green. The change ships.

By Thursday, a customer-success representative posts in Slack: “Has the agent changed? Tickets that used to escalate to a human are auto-resolving in a way that’s wrong”. The team checks the deploy history and sees nothing relevant - no model change, no library upgrade. They check the Sentry dashboard. Error rate is fine. Latency is fine. The agent is responding successfully to every request. From every dashboard the team owns, the system is healthy.

By the next Monday, three more customer-success messages have arrived. A senior engineer pulls the prompt change off the audit trail and reads the diff. The clarification of the category definition has shifted the agent’s threshold for “this is a billing question” downward; it is now classifying ambiguous requests as billing-resolvable when previously it would have escalated. The eval suite did not catch this because the golden dataset for “billing vs. escalate” had three examples and none of them sat near the new threshold.

The fix takes ninety minutes - revert the prompt to its prior version, redeploy, write a retroactive incident note. The damage takes longer. Several dozen customers got responses they should not have. The customer-success team spends a week unpicking the ones they can identify. The ones they cannot identify, they do not.

This was not a model regression. This was a code change shipped without the test that mattered. The team had treated the system prompt as a comment - text that lives next to code, not text that is code. There was no version of the prompt pinned in the eval suite. There was no canary that would have routed 5% of traffic through the candidate prompt and surfaced the threshold shift before the other 95%. There was no quality SLO that would have alerted on the escalation-rate change on Thursday morning. The prompt was treated as a configuration tweak; in fact it was a behavioral change to a customer-facing system, shipped without any of the controls the team would have required for an equivalent code change.

The discipline that would have prevented this is not exotic. Pin the prompt as a deployable artifact with a version. Add the boundary examples - the ones near the threshold - to the eval suite, not just the obvious ones. Route prompt changes through a canary the way you route schema changes. Set an SLO on escalation rate and alert when it drifts more than two percentage points over a rolling window. Each of these is a one-day task for an engineer who has shipped APIs. None of them are present in the team’s current operating model for the agent.

What was missing was not technology. It was the recognition that a prompt is a deployable artifact that affects customer-facing behavior, and therefore deserves the same treatment as any other customer-facing artifact. Without that recognition, prompts get edited like comments and shipped like config tweaks. With it, the failure mode is caught at PR time on Tuesday morning, not on Thursday afternoon when a customer notices.

The maintainer agent you ship in chapter 3 has the same shape: a system prompt, a five-label enum, and gray-zone fixtures your eval set does not cover until chapter 8. A “small clarity improvement” to that prompt is the same class of change as this Tuesday PR - customer-facing behavior with no boundary tests.

Chapter 6 establishes the output contract that pins the prompt’s expected behavior. Chapter 7 writes the prompt-snapshot test that fails CI when the prompt changes without an intentional eval update. Chapter 8 makes the eval suite the gate. Chapter 11 builds the rollback procedure. Chapter 12 puts the canary in front of every prompt change. Five chapters; one failure mode closed.

A team has a Pydantic AI agent running an internal customer-support tool. Cost-per-month at launch was forecast at $1,200, based on token-cost estimates from the eval set. After three months, the OpenAI invoice for the previous month is $4,800. Finance asks why. Engineering does not have an answer ready.

It takes the team four days to reconstruct what happened. The team had shipped two changes in the previous month: a new tool that fetches context from a third-party API, and a system-prompt update that encourages the agent to “verify your reasoning before responding”. Either change in isolation would have been fine. Together, they caused the agent to enter a longer tool-and-reasoning loop on roughly 30% of requests. Average tokens per request went from 1,800 to 6,400.

Chapter 2 already prints dollars for a fifty-issue classification run. This failure is what happens when nobody treats that line as a budget. Two weeks after the invoice arrives, a different signal surfaces: one customer is responsible for 18% of total agent traffic. They are not a power user - they have a buggy retry loop in their integration that triggers the agent four times for every actual user request. The agent gets called 40,000 times a month from one customer’s broken client. None of those calls produce billable outcomes for the team; they are pure cost. The team has no per-user cost telemetry, so this customer was indistinguishable from the rest of the traffic in every dashboard.

The two signals together explain the invoice. Each is an instance of the same gap: cost was treated as a number to read at the end of the month, not a metric to observe in real time.

This is the failure mode that destroys budget approval for the next agent. The CFO does not see “the team learned something”. The CFO sees “the team shipped a system whose costs they could not predict or control”. The next agent proposal goes through a different approval process, with different scrutiny, often with different ownership. The damage compounds beyond the immediate invoice.

The fix is not about being clever. It is about measuring the right thing. Cost-per-request is a first-class metric, derived from token counts and per-model pricing, exported on every span the same way latency is. Cost-per-user is a derived metric on top of cost-per-request, bucketed by whatever identifier the system already uses for authentication. A cost SLO - “cost-per-request must not exceed $0.04 at the 95th percentile” - runs in CI on the eval set, fails the PR when a change crosses it, and runs in production as a rolling alert. None of this is new engineering. It is the same observability discipline that already applies to latency and error rate, extended one more axis.

What was missing was not insight. It was the act of treating cost as a metric on equal footing with latency and error rate. Latency has a budget. Latency has alerts. Latency has a place in every postmortem. Cost should have all three. Without that, the system’s economics are a surprise every month - sometimes a pleasant one, often not.

Chapter 9 makes cost-per-issue a structured log field on every agent run and a span attribute on every trace. Chapter 10 builds the cost SLO and the cost regression gate in CI. Chapter 12 wires the cost SLO into production alerting. Three chapters; one failure mode closed.

These three failures are not the same incident. The retrieval miss is a measurement gap. The prompt change is a versioning and gating gap. The cost blowup is an observability gap. They look like three different problems.

They are not three different problems. They are three instances of the same problem: an artifact that affects production behavior was deployed without the discipline that would have applied to any other artifact that affected production behaviour.

The retriever’s chunk boundaries determined the system’s accuracy. They were treated as an implementation detail rather than as a configuration with a measurable quality and a versioned shape. The system prompt determined the agent’s classification threshold. It was treated as a piece of explanatory text rather than as a customer-facing artifact with a behavioral contract. The cost-per-request determined whether the system could be afforded at all. It was treated as a monthly report-out rather than as a metric with a budget.

In each case, the team had every tool they needed already. They had structured logging. They had a CI pipeline. They had dashboards and alerts. They had a postmortem template they had used for API regressions. They had pull-request reviews and version control. What they did not have was the recognition that all of these tools apply to the parts of the system that involve a model, with no modification beyond what they already ran for the parts that did not.

This is the entire thesis of this book. An LLM call is a non-deterministic IO boundary, no different in shape from a flaky third-party API. The discipline that already applies to flaky third-party APIs - versioned contracts, canaries, SLOs, alerts on tail latency, rollback procedures, postmortems with timelines - applies to LLM calls, translated into the right metric and the right artifact. Agents are software. The discipline you already have for shipping software is the discipline you need for shipping agents.

There is one framework where this translation is more natural than in any other framework available today. The rest of the chapter is about why.

In production engineering, a seam is the place in a system where you can substitute one implementation for another at test time without changing the rest of the system. Seams are how you write fast tests against a slow database. Seams are how you assert that your service handles a 5xx from a downstream API without ever making the downstream call. A system without seams is a system that can only be tested end-to-end, which is to say, a system that cannot be tested at all on a developer’s laptop.

Most LLM frameworks have very few seams. The agent is constructed dynamically, the prompt is templated at runtime, the tools are bound to the agent in a way that makes them hard to substitute, and the output is a string that the calling code parses with a regex or a try/except. None of those parts has a stable shape that a test can assert against. To test the system, you have to run the system. To run the system, you have to call the model. CI either becomes slow and flaky, or the team stops testing the parts that matter.

Pydantic AI is built around two seams that change this. The first is the typed output:

1 from pydantic import BaseModel, Field
2 from typing import Literal
3 
4 class IssueClassification(BaseModel):
5     label: Literal["bug", "feature-request", "question", "duplicate", "spam"]
6     confidence: float = Field(ge=0.0, le=1.0)
7     rationale: str

This is a Pydantic model the agent must produce. Every model response is validated against it before any of your code sees it. The seam is the model class itself: you can assert against it without running the LLM, you can version it the way you version an API response, you can write a test asserting that result.label == 'bug', and you can run that test against a deterministic stub of the agent in CI. The output stops being a string you parse and starts being a contract you test.

The second seam is dependency injection. Pydantic AI’s agent takes a deps parameter that is passed through to every tool. At production runtime, deps carries (for instance) a real GitHub client and a real database. In tests, deps carries fakes. Substituting them is one line:

1 result = await agent.run(
2     "triage this issue",
3     deps=Deps(github=FakeGitHubClient(), repo="octocat/hello-world"),
4 )

That single substitution is what makes the agent unit-testable. Without it, every test has to either hit the real GitHub or monkey-patch deep into the library. With it, a tool test is the same shape as any other test that uses a fake for a downstream service. The discipline you already have for testing API endpoints with a fake database connection is, without modification, the discipline you use for testing this agent with a fake GitHub.

If you have shipped LangChain to production, you already know which seams it does not give you. The same patterns are achievable there, with significantly more scaffolding - but the patterns sit uphill from the framework’s defaults rather than being the framework’s defaults themselves. The rest of this book leans on these two seams in every chapter. If they were not present, none of the chapters that follow are practical.

The agent that runs through every chapter is an open-source maintainer assistant. By the end of chapter 2 it classifies an incoming GitHub issue into a typed enum, running on your laptop against the last fifty issues of a real public repository. By the end of chapter 3 it is live on your own repo, posting comments on real issues via a CLI cron job. The classification will be rough. Every issue gets the same public comment whether confidence is 0.51 or 0.97. It will mislabel duplicates and gray-zone cases. None of that is a problem; it is the curriculum.

Every chapter from chapter 4 onward fixes exactly one of those reasons it is bad. Tools, the output contract, the test suite, the eval suite, observability, cost, human-in-the-loop, the canary, the runbook. Each chapter ends with the v0 agent fixed in one specific way, and a new failure mode visible that the next chapter addresses. By chapter 13, the agent is the same agent it was on chapter 3 - only now it survives traffic, prompt changes, and Monday morning.

The argument is done. Stop reading. Open your editor. Chapter 2 is waiting.