Chapter 02 - The First Agent

The argument is done. This chapter builds the thing.

By the end of this chapter, you have a Pydantic AI agent that reads a real GitHub issue and returns a validated classification: bug, feature-request, question, duplicate, or spam. It runs on your laptop against 50 closed issues from triage-lab, the book’s issues repository - curated fixtures with labels that match the agent’s output contract. It tells you where it disagrees with the human labels. It tells you what each classification costs.

None of that is impressive yet. It is the foundation that every subsequent chapter builds on. The agent you build here is the agent you will ship in chapter 3, observe breaking in chapter 4, and fix - systematically, one failure mode at a time - for the rest of the book.

Before writing a single line of agent code, write the output model. This is not a habit. It is a constraint that changes how you think about the problem.

 1 # models.py
 2 from typing import Literal
 3 
 4 from pydantic import BaseModel, Field
 5 
 6 
 7 class IssueClassification(BaseModel):
 8     label: Literal["bug", "feature-request", "question", "duplicate", "spam"]
 9     confidence: float = Field(ge=0.0, le=1.0)
10     rationale: str

Three fields. label is a Literal enum - not a string the model can fill however it likes, but a closed set of five values that Pydantic will enforce. confidence is a float bounded between 0 and 1. rationale is a string: the agent’s reasoning in one or two sentences, which you will use in chapter 8 when you build the eval suite.

That Literal is already doing production work. If the model returns "enhancement" or "bug report" or any value outside the five, Pydantic AI retries the model with a validation error - automatically, before your code ever sees the result. The output is either a valid IssueClassification or an exception. There is no third option. That is what a contract looks like.

You will version this model in chapter 6. You will write tests against it in chapter 7. You will run evals against it in chapter 8. All of that is possible because the output has a stable, typed shape. Start with the shape before anything else.

All runnable code for this chapter lives in chapter-02/ at the root of the book companion repository. Pydantic AI requires Python 3.10+; the project pins 3.12 via .python-version for reproducibility.

Issue data lives in triage-lab - 50 closed fixtures labeled with the book’s five categories, plus golden datasets for chapters 5 and 8. The default repo is nunombispo/triage-lab (read-only eval).

Install uv if you do not have it, then bootstrap the project:

1 cd chapter-02
2 uv sync                    # install pinned deps from uv.lock
3 cp .env.example .env       # then fill in keys

If you are creating the project from scratch instead of cloning the repo:

1 mkdir chapter-02 && cd chapter-02
2 uv init --name chapter-02 --no-readme
3 uv python pin 3.12
4 uv add "pydantic-ai-slim[anthropic]==1.97.0" "httpx>=0.28,<1" "python-dotenv>=1.0,<2"

pydantic-ai-slim[anthropic] pulls in only the Anthropic provider - enough for this chapter without installing every other model backend. Versions are pinned in pyproject.toml and uv.lock so your environment matches the book.

Create a .env file (or copy from .env.example):

1 ANTHROPIC_API_KEY=sk-ant-...
2 GITHUB_REPO=nunombispo/triage-lab
3 GITHUB_TOKEN=ghp_...                # optional for classify.py; raises rate limit

The GitHub token is optional for 50 issues but worth having. It takes 30 seconds to generate at github.com/settings/tokens with no scopes required for public repo read access.

 1 # agent.py
 2 from dotenv import load_dotenv
 3 from pydantic_ai import Agent
 4 
 5 from models import IssueClassification
 6 
 7 load_dotenv()
 8 
 9 SYSTEM_PROMPT = """You are an open-source issue classifier. Given a GitHub issue \
10 title and body, classify it into exactly one of these categories:
11 
12 - bug: reports unexpected behavior or a clear defect
13 - feature-request: asks for new functionality or a change in behavior
14 - question: asks how to do something or seeks clarification
15 - duplicate: the same issue already exists (use only when certain)
16 - spam: irrelevant, automated, or off-topic content
17 
18 Return your confidence as a float between 0.0 and 1.0. Return your rationale \
19 in one or two sentences explaining what signals drove the classification."""
20 
21 agent = Agent(
22     "anthropic:claude-sonnet-4-6",
23     output_type=IssueClassification,
24     system_prompt=SYSTEM_PROMPT,
25 )

Four things to notice.

"anthropic:claude-sonnet-4-6" is the model string — Claude Sonnet 4.6, a capable mid-tier Anthropic model. Pydantic AI uses provider:model-name format. To switch to a cheaper model, replace it with "anthropic:claude-haiku-4-5". To switch to OpenAI, use "openai:gpt-4o-mini". That is the entire migration. The agent.run() call, the output model, the system prompt — none of it changes, because the model is a configuration string, not something wired into your code. Call this the model-agnostic interface. This is what it looks like in practice.

output_type=IssueClassification is the seam chapter 1 called the typed output. The agent is not instructed to “return JSON with a label field.” It is handed a Pydantic model and told to satisfy it. Pydantic AI handles the serialization, the validation, and the retry on validation failure. Your code works with a typed object, not with a string.

The system prompt names the five categories explicitly and distinguishes the edge cases that matter — “use duplicate only when certain” is doing real work. The model’s behavior on ambiguous inputs depends on the specificity of these instructions. Chapter 7 will show you how to test this specificity without calling the model.

rationale in the output model is not decoration. You will query it in chapter 8 when an LLM judge needs to evaluate whether the classification was reasonable. Put it in the model now even though you will not use it until later. The cost is negligible. The cost of retrofitting it later is not.

 1 # fetch_issues.py
 2 import os
 3 from typing import Optional
 4 
 5 import httpx
 6 
 7 DEFAULT_REPO = "nunombispo/triage-lab"
 8 
 9 
10 def fetch_closed_issues(
11     repo: Optional[str] = None,
12     limit: int = 50,
13     token: Optional[str] = None,
14 ) -> list[dict]:
15     repo = repo or os.getenv("GITHUB_REPO", DEFAULT_REPO)
16     headers = {"Accept": "application/vnd.github+json"}
17     if token:
18         headers["Authorization"] = f"Bearer {token}"
19 
20     issues: list[dict] = []
21     page = 1
22     while len(issues) < limit:
23         response = httpx.get(
24             f"https://api.github.com/repos/{repo}/issues",
25             params={
26                 "state": "closed",
27                 "per_page": 100,
28                 "page": page,
29             },
30             headers=headers,
31             timeout=10.0,
32         )
33         response.raise_for_status()
34         page_items = response.json()
35         if not page_items:
36             break
37         batch = [i for i in page_items if "pull_request" not in i]
38         issues.extend(batch)
39         page += 1
40 
41     return issues[:limit]

The default repo is triage-lab — fixtures with labels that match the agent’s five categories exactly.

The GitHub Issues API returns pull requests alongside issues unless you filter them — "pull_request" not in i drops them. The state=closed filter gives you issues that have human labels applied and resolved, which is what you want for comparing against.

 1 # classify.py
 2 import asyncio
 3 import os
 4 
 5 from dotenv import load_dotenv
 6 from pydantic_ai.exceptions import ModelHTTPError
 7 
 8 load_dotenv()
 9 
10 from agent import agent
11 from fetch_issues import fetch_closed_issues
12 
13 RETRYABLE_STATUS = {429, 502, 503, 504, 529}
14 MAX_ATTEMPTS = 5
15 
16 
17 async def run_agent_with_retry(prompt: str):
18     for attempt in range(1, MAX_ATTEMPTS + 1):
19         try:
20             return await agent.run(prompt)
21         except ModelHTTPError as e:
22             if e.status_code not in RETRYABLE_STATUS or attempt == MAX_ATTEMPTS:
23                 raise
24             wait = min(2**attempt, 60)
25             print(
26                 f"  API {e.status_code} ({e.model_name}), retry in {wait}s "
27                 f"({attempt}/{MAX_ATTEMPTS})...",
28                 flush=True,
29             )
30             await asyncio.sleep(wait)
31 
32 
33 async def classify_issue(issue: dict) -> dict:
34     text = f"Title: {issue['title']}\n\nBody:\n{(issue['body'] or '')[:1500]}"
35     result = await run_agent_with_retry(text)
36     usage = result.usage
37     return {
38         "number": issue["number"],
39         "title": issue["title"],
40         "human_labels": [label["name"] for label in issue["labels"]],
41         "agent_label": result.output.label,
42         "confidence": result.output.confidence,
43         "rationale": result.output.rationale,
44         "tokens_input": usage.input_tokens,
45         "tokens_output": usage.output_tokens,
46     }
47 
48 
49 async def main() -> None:
50     token = os.getenv("GITHUB_TOKEN")
51     issues = fetch_closed_issues(token=token)
52     print(f"Fetched {len(issues)} issues\n")
53 
54     total_input = 0
55     total_output = 0
56     for issue in issues:
57         r = await classify_issue(issue)
58         total_input += r["tokens_input"]
59         total_output += r["tokens_output"]
60         match = "✓" if r["agent_label"] in r["human_labels"] else "✗"
61         print(
62             f"{match} #{r['number']:5d}  {r['agent_label']:16s}  "
63             f"({r['confidence']:.2f})  {r['title'][:55]}"
64         )
65 
66     # Sonnet 4.6 list prices: $3/MTok input, $15/MTok output
67     cost = (total_input * 3.00 + total_output * 15.00) / 1_000_000
68     print(f"\n{len(issues)} issues — {total_input + total_output:,} tokens — ${cost:.4f}")
69 
70 
71 if __name__ == "__main__":
72     asyncio.run(main())

Run it from chapter-02/:

1 uv run classify.py

The output will look roughly like this (token counts and cost depend on the model and responses; this sample is from a full 50-issue run on Sonnet 4.6):

 1 Fetched 50 issues
 2 
 3 ✓ #   50  question          (0.46)  Field marked optional still required at runtime?
 4 ✓ #   49  feature-request   (0.82)  Document OpenAPI export workflow end-to-end
 5 ...
 6 ✓ #   35  bug               (0.95)  UUID4 field accepts UUID1 values without error
 7 ✓ #   34  duplicate         (0.72)  EmailStr validation fails for IDN email addresses
 8 ✓ #   33  bug               (0.38)  EmailStr rejects plus-addressing with unicode local par
 9 ✓ #   32  duplicate         (0.92)  JSON parsing fails on trailing comma
10 ✗ #   31  feature-request   (0.60)  Cannot parse JSON with trailing comma in strict mode
11 ✗ #   30  bug               (0.82)  Behavior differs from documentation for validate_assign
12 ✓ #   29  question          (0.52)  Validation 3x slower after upgrade — is this a bug?
13 ✓ #   28  question          (0.95)  Migrating `@validator` with `always=True` to v2?
14 ✓ #   27  question          (0.93)  Best practice for generic BaseModel subclasses?
15 ✓ #   26  question          (0.93)  Are async field validators supported?
16 ✓ #   25  question          (0.95)  model_config extra='forbid' vs Field on every attribute
17 ...
18 ✓ #    2  bug               (0.97)  Segfault when validating deeply nested model (>200 leve
19 ✓ #    1  question          (0.52)  model_validator runs twice when using inheritance
20 
21 50 issues — 51,534 tokens — $0.2331

About twenty-three cents for the full run — Sonnet 4.6 at $3 per million input tokens and $15 per million output tokens. Your token split will differ run to run; the total line will move with it. Keep that order of magnitude in your head; you will come back to it in chapter 10.

Scroll the output and find the ✗ lines before you read the next section. On this run there are two.

On a full 50-issue run, most lines show ✓. Two ✗ lines are worth stopping on — they disagree for different reasons.

1 ✗ #   30  bug               (0.82)  Behavior differs from documentation for validate_assign

The fixture gray-zone-docs-mismatch title: “Behavior differs from documentation for validate_assignment.” The body: “Docs say assignment validates. Setting attribute on frozen model raises ValidationError but value still changes in __dict__. Documentation bug or implementation bug?”

The agent labeled bug at 0.82. The human labeled question.

The agent read “behavior differs from documentation” as a defect report — unexpected behavior against the spec. The human read the same issue as a triage question: the reporter is not sure whether the bug is in the docs or the code. The closing sentence is literally asking which bucket it belongs in.

Both are defensible. This is the gray zone between “I found a bug” and “I need a maintainer to tell me what kind of problem this is.” At 0.82 confidence, the agent is overclaiming. An honest read would land closer to 0.55 and let escalation logic route it to a human.

1 ✗ #   31  feature-request   (0.60)  Cannot parse JSON with trailing comma in strict mode

The fixture dup-original-trailing-comma title: “Cannot parse JSON with trailing comma in strict mode.” The body: “Input {\"a\": 1,} fails validation. Standard JSON forbids trailing comma. Need opt-in lenient JSON mode.”

The agent labeled feature-request at 0.60. The human labeled bug.

The agent latched onto “Need opt-in lenient JSON mode” — that is a request for new behavior. The human labeled bug because the report starts from a concrete failure: strict parsing rejects input the reporter’s API already sends. Same underlying issue as #32 (duplicate), which the agent got right at 0.92.

Here the agent is closer to the right confidence — 0.60 signals uncertainty — but the label disagreement is structural, not ambiguous. Your enum forces every issue into one of five categories; “fix the parser” and “add a lenient mode flag” are different maintainer actions. A human triager might file this as bug first and open a design discussion for lenient JSON as a follow-up.

This is the classification problem no system prompt will fully solve. Some issues are genuinely ambiguous (#30). Others sit on a boundary between categories that your contract treats as mutually exclusive (#31). The agent’s job is not to eliminate that tension — it is to classify consistently and use confidence to flag when a human should look.

That is what the eval suite in chapter 8 is for. You will build a golden dataset, which marks both fixtures as gray_zone or duplicate pairs — and measure whether the agent’s confidence is calibrated. Not “does it get the right label” but “does it know when it does not know.”

For now, note the disagreements. On #30 the agent may have the wrong label but the wrong confidence is the bigger problem. On #31 the label fight is real and the 0.60 score is doing its job. Those are different failures with different fixes.

The four files total about 80 lines of Python. What they give you:

A working classification agent that validates its output against a typed contract, runs against a real repository’s issues, and tells you exactly what it costs to operate.

What they do not give you, yet: tests, tools, observability, or a way to run outside your laptop. That is intentional.

The agent classifies issues. On this first full run it agreed with human labels on 48 of 50 fixtures — the two ✗ lines above. That number will move when you change the model, tune the system prompt, or point at a noisier repo. Even two disagreements are enough to see where the agent overclaims or picks the wrong category. You do not need a high error rate before chapter 3; you need real failures you can name.

Open chapter 3. Ship it.