Leanpub Header

Skip to main content

Claude Code: Building Production Agents That Actually Scale

Engineer governed Claude Code agents that survive real production failures

The instructor has published 92% of this course.Last updated on 2026-09-01

Move beyond the demo and build a Claude Code agent your team can operate. You will implement tool boundaries, evaluation gates and recovery paths, then assemble the evidence for a production-readiness review.

Minimum price

$49.00

$149

You pay

Author earns

$

Also available for 1 course credit with a Learner Membership

PDF
EPUB
WEB
100,926Words
About

About

About the Course

Build a Claude Code agent that your team can test, govern and recover when something goes wrong. This practical course takes you from the agent loop to a production-readiness dossier: the controls, tests and operational evidence needed to review an agent before release.

You will design tool and permission boundaries, manage context and memory, evaluate model routing, work with MCP and hooks, and test failure and recovery paths. Practical labs ask you to implement controls and record evidence, not just follow a successful demo.

Across eight modules, the written lessons, worked examples, labs and assessments help you assemble a routing policy, context controls, evaluation cases, an observability plan, a recovery procedure and release evidence. Video material supports the course; a video is not promised for every lesson.

This is the hands-on course companion to Thomas De Vos's book, Claude Code: Building Production Agents That Actually Scale. The book is a separate product and an optional reference, not a required purchase. This course focuses on applying the ideas through practical work and assessments.

For software engineers, technical leads, platform engineers, solution architects and security engineers. You should be comfortable with a terminal, Git, configuration files, tests and normal software-delivery workflows. Prior experience with the Claude Agent SDK is not required.

Leanpub offers a 60-day refund policy, subject to its course exclusions: a course is not refundable after you generate its certificate or fail its last or sole permitted attempt. Read the current policy at leanpub.com/refunds before purchasing.

About the instructor

Thomas De Vos is an architect and technical author with more than 25 years of experience. He is a Google Cloud Generative AI Leader and Google Cloud Authorised Trainer. His teaching focuses on the engineering decisions, practical controls and evidence needed to operate AI systems responsibly.

Free sample: CC-001 lesson and practical lab

This written sample contains the first lesson and its practical lab. The course video, worksheet download and following knowledge-check assessment are not included in this sample. For the lab, use a blank document as your Agent Loop Canvas and record each boundary listed in the instructions.

CC-001: The Agent Loop, Explained and Enforced

An agent is not a clever answer. It is a sequence of decisions and side effects, with evidence and control at every boundary.

Estimated lesson time: 38 min

Learning objectives

  • Trace a Claude Code task as model decisions, tool requests, observations and termination decisions.
  • Distinguish a proposed tool call from an authorized and completed side effect.
  • Design pre-action validation using schema checks, permissions, policy, sandboxing, state preconditions and approval.
  • Verify postconditions and define stop, retry and escalation behaviour for a bounded run.

Start with the loop, not the chat transcript

Claude Code wraps a language model in an agentic harness. The model receives the current prompt, conversation, instructions and tool definitions. It can answer in text or request a tool. Claude Code executes an authorized request, returns the result to the model, and the model decides what to do next. The loop ends when the model produces a final response without another tool request, or when the runtime stops it.

The useful production unit is therefore not one answer. It is a chain of state transitions. Each transition should have an input, a proposed action, an authorization decision, an observed result and a new state. If the team stores only the final prose, it cannot reliably explain what changed or why.

Anthropic's Agent SDK documentation calls each model-and-tool round trip a turn. A single task can use many turns. In the example below, reading a file changes only the agent's knowledge; editing a file changes repository state; running tests produces new evidence; the final answer changes neither.

  • Turn: 1. Model decision: Inspect manifests and repository rules. Runtime event: Read tools return CLAUDE.md and package files. State after the event: The agent knows scope, current versions and constraints.
  • Turn: 2. Model decision: Propose a manifest edit. Runtime event: Permission and policy checks run before Edit. State after the event: Either no change, an approval request, or a tracked file change.
  • Turn: 3. Model decision: Verify the candidate patch. Runtime event: Approved test command runs in the sandbox. State after the event: A test result and artifact become new evidence.
  • Turn: 4. Model decision: Finish or repair. Runtime event: No tool call means the loop returns a result. State after the event: The run ends with status, evidence and remaining uncertainty.

Common misconception

The model does not directly 'use the terminal'. It requests a tool call. The Claude Code runtime, permissions, hooks and operating environment determine whether and how that call executes.

Validation is a stack of different questions

The earlier line - 'validate the proposed action before the side effect occurs' - was directionally right but educationally incomplete. Validation is not one magic function. It is a sequence of gates, and each gate answers a different question. Passing one gate does not imply the others passed.

Run deterministic checks before judgement-based ones. A malformed tool request should fail schema validation before it reaches a policy model or a human. A forbidden path should be denied by policy even if the patch itself looks correct. A permitted action can still need approval because the consequence is high.

  1. Schema: is the tool name known, and are required fields present with valid types and normalized paths?
  2. Authorization: is this tool and operation allowed, denied or always subject to approval for this project and identity?
  3. Policy: does the target, command, data class, branch, amount or destination satisfy business and security rules?
  4. Precondition: is the world still in the state on which the proposal was based - for example, the expected Git revision and a clean working tree?
  5. Containment: can the command run inside the configured filesystem and network sandbox without broader host access?
  6. Approval: if policy says 'ask', does an accountable person receive enough evidence to approve this exact action?
  7. Execution and postcondition: after the side effect, did the intended state change occur, and did forbidden state remain unchanged?

Production note

A validator should return an explicit outcome: allow, deny, ask, or - where the integration supports it - rewrite a narrow input. It should also record the rule, evidence and reason.

How Claude Code enforces the pre-action boundary

Claude Code permissions provide allow, ask and deny rules. Deny takes precedence over ask, and ask over allow. Allow means pre-approved, not exclusive: a tool that is absent from the allow list can still follow the active permission flow. To remove or universally block capability, use deny rules or omit the tool from an SDK run.

A PreToolUse hook runs before the normal permission decision and can inspect a proposed call. This is the right place for controls that must evaluate every matching tool request, such as approved file paths or command patterns. A canUseTool callback in the Agent SDK handles unresolved approvals, but it is not a universal enforcement point because earlier rules can resolve a request first.

Sandboxing is a separate boundary. On supported systems it constrains Bash filesystem and network access using operating-system enforcement. It does not replace permissions, and it does not sandbox built-in Read, Edit or Write tools. Production design normally needs both.

Example: Illustrative project policy in .claude/settings.json

{
  "permissions": {
    "allow": [
      "Read",
      "Grep",
      "Bash(npm test *)"
    ],
    "ask": [
      "Edit",
      "Write",
      "Bash(git push *)"
    ],
    "deny": [
      "Read(./.env)",
      "Read(./secrets/**)",
      "Bash(rm -rf *)"
    ]
  },
  "sandbox": {
    "enabled": true,
    "failIfUnavailable": true
  }
}

Common misconception

A CLAUDE.md sentence such as 'never edit deployment files' is useful guidance, but it is context, not hard enforcement. Put non-negotiable restrictions in permissions, hooks, tool wrappers or the sandbox.

A worked validator for a dependency-update agent

Suppose the agent may propose changes to package.json and package-lock.json, run npm test, and prepare a draft pull request. It may not edit workflow files, read secrets, push to the default branch or merge. The model requests an Edit to .github/workflows/release.yml because a newer action version appears necessary.

A strong system does not ask whether the model sounds confident. It evaluates the action against the contract. The normalized path is outside the write allow-list, so the request is denied before execution. The denial becomes an observation in the loop: the agent can revise its plan, explain that the workflow change requires a different authorized process, or stop.

Example: Validator decision record - not an LLM opinion

{
  "decision": "deny",
  "tool": "Edit",
  "target": ".github/workflows/release.yml",
  "rule": "manifest-paths-only",
  "reason": "target is outside the approved write set",
  "observed_revision": "8f31c2a",
  "side_effect_occurred": false
}
Worked example: Evaluate one proposed Edit

Proposed action: Edit .github/workflows/release.yml at repository revision 8f31c2a.

  1. Schema passes: Edit has a path and replacement content.
  2. Path normalization resolves the target inside the repository; no traversal or symlink escape is accepted.
  3. Authorization fails: the task contract permits writes only to package.json and package-lock.json.
  4. The runtime records deny, rule=manifest-paths-only, proposal hash and current revision. No edit occurs.
  5. Claude receives the denial reason and can propose an in-scope patch or escalate the workflow-file change to a human-owned task.

Result: The run remains useful without granting the model broader authority. A denied action is a controlled branch of the loop, not a runtime failure.

Postcondition checks prove what actually happened

Pre-action validation proves only that an attempt is permitted under the observed state. It does not prove success. A command can return an error after partially changing state; a network timeout can leave an external action completed but unconfirmed; a test can pass because the agent edited the test rather than fixed the product.

After each side effect, re-read the affected state and compare it with explicit postconditions. For a patch, inspect the diff, confirm only approved files changed, run the required immutable checks and retain their exit codes. For an external create operation, reconcile by idempotency key before retrying. The tool result should distinguish success, failure and unknown - not collapse all three into prose.

  • Action: Edit manifest. Precondition: Expected revision; approved target. Postcondition: Diff contains only intended dependency fields. On uncertainty: Re-read file and diff; do not claim success.
  • Action: Run tests. Precondition: Approved command and isolated environment. Postcondition: Required suite ran; exit status and artifacts captured. On uncertainty: Mark verification incomplete.
  • Action: Create draft PR. Precondition: Authenticated bounded identity; branch exists. Postcondition: PR ID exists and is draft; no merge occurred. On uncertainty: Query by idempotency key before retry.
  • Action: Finish run. Precondition: Acceptance criteria evaluated. Postcondition: Result contains status, evidence and unresolved items. On uncertainty: Return incomplete or escalated, not 'done'.

Common misconception

Exit code zero is evidence for a particular command, not proof that the whole task is correct. Verification must match the acceptance criteria and threat model.

Stopping is part of the design

An agent loop needs more than a success condition. Define maximum tool-use turns, elapsed time, spend, retries per action, repeated-state detection and escalation conditions. In the Agent SDK, maxTurns bounds tool-use round trips and maxBudgetUsd bounds estimated spend. When a limit is reached, the result reports a specific error subtype rather than pretending the task finished.

A useful termination policy distinguishes completed, safely incomplete, denied, escalated and failed. Stop when acceptance criteria pass; when the next required action is forbidden; when critical evidence is missing; when two attempts return the same failure without new information; or when a budget is exhausted. Continuing to 'try harder' without a state change is not autonomy. It is an unbounded loop.

Production note

The production loop is: observe → decide → propose → authorize → execute → verify → update state → continue or stop. Every arrow needs an owner and evidence.

Practical lab

Design the boundary for one side effect

Choose one real action - edit a file, create a ticket, send a message or open a pull request. Write its schema checks, allow/ask/deny policy, state preconditions, sandbox or tool boundary, approval packet, postconditions, idempotency rule and stop behaviour. Then test one allowed, one denied and one ambiguous outcome.

Deliverable: A completed Agent Loop Canvas plus three decision traces showing allow, deny and unknown outcomes.

Instructor

About the Instructor

Material

Course Material

  • Claude Code: Building Production Agents That Actually Scale

  • How to use this course

  • Prerequisites

  • Evidence standard

  • Video availability

  • Module 1 outcome

  • Module 1: Foundations of a production agent

  • Module map

  • CC-001: The Agent Loop, Explained and Enforced

  • Learning objectives

  • Start with the loop, not the chat transcript

  • Validation is a stack of different questions

  • How Claude Code enforces the pre-action boundary

  • A worked validator for a dependency-update agent

  • Worked example: Evaluate one proposed Edit

  • Postcondition checks prove what actually happened

  • Stopping is part of the design

  • Practical lab

  • Design the boundary for one side effect

  • CC-001 knowledge check

  • Review notes

  • Key takeaway

  • Sources and further reading

  • CC-002: What a Governed Claude Code Run Looks Like

  • Learning objectives

  • First, separate the terms that teams often blur

  • A governed run has an execution contract

  • Choose the operating surface deliberately

  • Worked example: Evolve one dependency workflow without jumping surfaces too early

  • Concrete example: a reproducible headless review

  • Worked example: What the wrapper must decide

  • Concrete example: the Agent SDK as a controlled host

  • The run record is the operational product

  • Follow one governed run from acceptance to evidence

  • Failure is a designed state, not an exception to governance

  • Practical lab

  • Specify one governed run end to end

  • CC-002 knowledge check

  • Review notes

  • Key takeaway

  • Sources and further reading

  • CC-003: Choosing the Right Model Through Workload Classes

  • Learning objectives

  • There is no universal list of classes - so define yours explicitly

  • A practical four-class starting taxonomy

  • Every class needs seven contract fields

  • Classify on consequence and evidence, not prompt difficulty

  • Worked example: Classify three operations in one service assistant

  • Choose a model path only after the class is known

  • Fallback is a fresh policy decision

  • Practical lab

  • Define four routing contracts for one product

  • CC-003 knowledge check

  • Review notes

  • Key takeaway

  • Sources and further reading

  • CC-004: Context Is a Budget, Not Memory

  • Learning objectives

  • Context engineering is an evidence pipeline

  • The context window is working memory, not a database

  • Allocate the budget before retrieval fills it

  • Worked example: Budget a dependency-update run

  • Authority, relevance, freshness and specificity resolve conflict

  • Compression must preserve decisions, uncertainty and provenance

  • Implement the context pipeline in Claude Code

  • Worked example: What the assembler does before Claude sees a token

  • Test and observe the context pipeline as a production component

  • Worked example: Diagnose a context failure instead of blaming the model

  • Practical lab

  • Engineer a context pipeline for one risky decision

  • CC-004 knowledge check

  • Review notes

  • Key takeaway

  • Sources and further reading

  • CC-005: The Production Agent Definition

  • Learning objectives

  • A demo proves capability; production proves controlled operation

  • The five properties are system properties

  • Bounded means the blast radius is enforced before action

  • Worked example: Bound a dependency-update agent

  • Observable means reconstructable, not merely verbose

  • Reversible requires stop, restore and compensate

  • Evaluable means release thresholds cover outcomes and process

  • Worked example: Create a release gate that a demo cannot game

  • Governable means someone can authorize, change and stop the system

  • Turn the scorecard into a release decision

  • Worked example: Score the dependency-update candidate

  • Ship through staged exposure and operate the learning loop

  • Practical lab

  • Run an evidence-based production readiness review

  • CC-005 knowledge check

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Module 1 Capstone - Production Agent Design Brief

  • Scenario

  • Assignment

  • Acceptance criteria

  • Rubric

  • Submission package

  • Module 1 graded assessment

  • Module 1 graded assessment

    3 attempts allowed

  • Assessment review guide

  • Module 1 resources

  • Module 1 Learner Workbook

  • Agent Loop Canvas

  • Operating Surface Decision Matrix

  • Model Routing Policy

  • Context Budget Worksheet

  • Production Readiness Scorecard

  • Capstone Design Brief

  • Module 1 Learner README

  • Capstone Rubric

  • Capstone solution guidance

  • Module 1 capstone solution guidance

  • Loop evidence

  • Operating-surface reasoning

  • Routing evidence

  • Context evidence

  • Readiness evidence

  • Common weak answers

  • Final self-review

  • Module 2 overview: The primitive stack

  • Outcomes

  • Lesson path

  • CC-006: Tools as governed API contracts

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: repository issue triage tool

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-006 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-007: Subagents and controlled delegation

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: parallel repository review

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-007 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-008: Hooks as deterministic control points

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: command policy hook

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-008 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-009: Skills as reusable operational knowledge

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: incident triage skill

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-009 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-010: MCP as the integration plane

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: read-only work tracking server

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-010 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-011: Plugins and trusted distribution

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: engineering review plugin

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-011 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-012: Assembling the primitive stack

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: dependency review agent

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-012 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • Module 2 capstone: Governed dependency review stack

  • Scenario

  • Assignment

  • Acceptance criteria

  • Rubric

  • Submission package

  • Module 2 graded assessment

  • Module 2 graded assessment

    3 attempts allowed

  • Assessment review guide

  • Module 2 resources

  • Tool contract worksheet

  • Delegation contract

  • Hook test matrix

  • Skill authoring checklist

  • MCP threat model

  • Plugin release checklist

  • Primitive stack canvas

  • Module 2 capstone template

  • Module 2 capstone rubric

  • Module 2 capstone solution guidance

  • Strong control path

  • Expected boundaries

  • Weak answers

  • Final review

  • Module 3 overview: Owning the runtime

  • Outcomes

  • Lesson path

  • CC-013: Choosing between CLI and Agent SDK

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: durable review queue

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-013 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-014: Building a minimal SDK agent

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: repository evidence service

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-014 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-015: Tools, hooks and subagents in the SDK

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: governed change reviewer

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-015 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • CC-016: Sessions, state and durability

  • Learning objectives

  • Mechanism

  • Implementation and configuration

  • Worked example: review that survives worker failure

  • Reading the evidence

  • Release reasoning

  • Failure modes

  • Practical lab

  • Evidence criteria

  • CC-016 formative exercise

  • Review notes

  • Key takeaway

  • Sources and further reading

  • Next step

  • Module 3 capstone: Durable review service

  • Scenario

  • Assignment

  • Acceptance criteria

  • Rubric

  • Submission package

  • Module 3 graded assessment

  • Module 3 graded assessment

    3 attempts allowed

  • Assessment review guide

  • Module 3 resources

  • SDK decision record

  • SDK runtime contract

  • Message handling matrix

  • Custom tool schema review

  • SDK hook test matrix

  • Subagent isolation plan

  • Session state schema

  • Recovery drill

  • Module 3 capstone template

  • Module 3 capstone rubric

  • Module 3 capstone solution guidance

  • Strong architecture decision

  • Expected runtime design

  • Expected recovery design

  • Weak answers

  • Final review

  • Module 4: Secure and govern the agent boundary

  • Module outcomes

  • Completion route

  • Evidence standard

  • Time plan

  • CC-017: Enforce permissions as runtime policy

  • Learning objectives

  • Why this lesson matters

  • Mechanism

  • 1. Global interaction mode and per-tool policy are separate controls

  • 2. Default deny starts with no capabilities and adds only proved requirements

  • 3. Path, command, method, endpoint, time and budget limits belong at execution boundaries

  • 4. Denied actions are useful evidence when they are structured and observable

  • Implementation

  • Worked example: A release agent needs to read source, write a patch, run a fixed test command and create a draft change request

  • Failure modes

  • Operating review

  • Practical lab

  • Build and test your permissions boundary

  • CC-017 formative exercise

  • Evidence criteria

  • Sources and further reading

  • Bridge

  • CC-018: Contain agents with measurable sandboxes

  • Learning objectives

  • Why this lesson matters

  • Mechanism

  • 1. Read, write, execute and transmit are the four blast-radius dimensions

  • 2. Application policy, operating-system controls and infrastructure isolation must agree

  • 3. Read-only access can still create severe disclosure risk through logs or egress

  • 4. A sandbox is credible only after realistic escape and recovery tests

  • Implementation

  • Worked example: A document-review agent receives untrusted archives

  • Failure modes

  • Operating review

  • Practical lab

  • Build and test your sandbox boundary

  • CC-018 formative exercise

  • Evidence criteria

  • Sources and further reading

  • Bridge

  • CC-019: Control egress, secrets and data movement

  • Learning objectives

  • Why this lesson matters

  • Mechanism

  • 1. Egress is an authorization decision, not a networking convenience

  • 2. Credentials should be injected at the narrowest component that needs them

  • 3. Classification must drive context, output, logging, retention and deletion rules

  • 4. Seeded canaries make accidental disclosure detectable during tests

  • Implementation

  • Worked example: A case-triage agent calls an internal records API and a specialist reference service

  • Failure modes

  • Operating review

  • Practical lab

  • Build and test your data boundary

  • CC-019 formative exercise

  • Evidence criteria

  • Sources and further reading

  • Bridge

  • CC-020: Operate policy, audit and lineage as code

  • Learning objectives

  • Why this lesson matters

  • Mechanism

  • 1. Policy must be versioned, reviewed, validated and pushed by deployment

  • 2. Effective policy includes base rules plus environment and workload overlays

  • 3. Audit records capture proposed, authorized, denied and completed actions without secret values

  • 4. Lineage joins request, context, model, policy, tool calls, approvals and output

  • Implementation

  • Worked example: A release investigation asks why a repository write occurred

  • Failure modes

  • Operating review

  • Practical lab

  • Build and test your governance record

  • CC-020 formative exercise

  • Evidence criteria

  • Sources and further reading

  • Bridge

  • CC-021: Govern plugins, skills and MCP dependencies

  • Learning objectives

  • Why this lesson matters

  • Mechanism

  • 1. An extension inherits the useful access of the process that hosts it

  • 2. A friendly name and active maintainer do not establish provenance

  • 3. Versions and transitive dependencies must be pinned by digest

  • 4. Admission is temporary and needs expiry, monitoring and a tested revoke path

  • Implementation

  • Worked example: A team proposes an MCP server for issue search

  • Failure modes

  • Operating review

  • Practical lab

  • Build and test your supply-chain gate

  • CC-021 formative exercise

  • Evidence criteria

  • Sources and further reading

  • Bridge

  • Module 4 capstone: Boundary assurance packet

  • Scenario

  • Brief

  • Required submission

  • Execution requirements

  • Scoring rubric

  • Submission checklist

  • Module 4 graded assessment

  • Module 4 graded assessment

    3 attempts allowed

  • Assessment review guide

  • Module 4 resources

  • Downloads

  • Visuals

  • Use and retention

  • Module 4 solution guidance

  • Strong submission pattern

  • Common weak answers

  • Review sequence

  • Model decision

  • Self-scoring

  • Module 5: Evaluate agent behavior before and after release

  • Module outcomes

  • Completion route

  • Evidence standard

  • Time plan

  • CC-022: Build an eval-first release system

  • Learning objectives

  • Why this lesson matters

  • Mechanism

  • 1. Outcome, process and cost measures answer different release questions

  • 2. Tasks, trials and transcripts are distinct units and must not be conflated

  • 3. Fast pull-request gates and broader scheduled suites serve different feedback loops

  • 4. Threshold breaches need precommitted alert, rollback or stop responses

  • Implementation

  • Worked example: A change-review agent appears accurate but doubles tool calls and sometimes reads unrelated files

  • Failure modes

  • Operating review

  • Practical lab

  • Build and test your evaluation contract

  • CC-022 formative exercise

  • Evidence criteria

  • Sources and further reading

  • Bridge

  • CC-023: Design trajectory evals and golden datasets

  • Learning objectives

  • Why this lesson matters

  • Mechanism

  • 1. A golden case includes input, acceptable path constraints, result and expert rationale

  • 2. The agent under test must not access labels before its run completes

  • 3. Model, effort, tools, time, data and randomness must be pinned or recorded

  • 4. Small sets need stratification, repeated trials and uncertainty-aware reporting

  • Implementation

  • Worked example: A code-review agent finds a real defect after reading an unrelated secret file

  • Failure modes

  • Operating review

  • Practical lab

  • Build and test your trajectory harness

  • CC-023 formative exercise

  • Evidence criteria

  • Sources and further reading

  • Bridge

  • CC-024: Calibrate LLM judges for production gates

  • Learning objectives

  • Why this lesson matters

  • Mechanism

  • 1. A judge has no privileged ground truth and must be treated as a fallible evaluator

  • 2. Specific criteria, anchored contrasts and evidence requirements improve discrimination

  • 3. Agreement corrected for chance matters more than raw agreement alone

  • 4. High-impact disagreement routes to humans and can never be averaged away

  • Implementation

  • Worked example: Three domain reviewers label fifty release summaries

  • Failure modes

  • Operating review

  • Practical lab

  • Build and test your judge gate

  • CC-024 formative exercise

  • Evidence criteria

  • Sources and further reading

  • Bridge

  • Module 5 capstone: Evaluation release gate

  • Scenario

  • Brief

  • Required submission

  • Execution requirements

  • Scoring rubric

  • Submission checklist

  • Module 5 graded assessment

  • Module 5 graded assessment

    3 attempts allowed

  • Assessment review guide

  • Module 5 resources

  • Downloads

  • Visuals

  • Use and retention

  • Module 5 solution guidance

  • Strong submission pattern

  • Common weak answers

  • Review sequence

  • Model decision

  • Self-scoring

  • Module 6: Operate What You Ship: Observability, Reliability and Cost

  • Outcomes

  • Lesson map

  • Completion standard

  • CC-025: Instrumenting the Agent Timeline

  • Learning objectives

  • Mechanism: turn execution into correlated evidence

  • Implementation sequence

  • What to instrument

  • Build the trace hierarchy

  • Assign each observability layer one job

  • Instrument a tool boundary

  • Separate experiments from production monitoring

  • Add an agent-aware debugging view

  • Operate through a production monitoring backend

  • Design dashboards around operator questions

  • Connect service indicators to automatic responses

  • Page only for an actionable condition

  • Worked example: instrument a screening workflow

  • Worked example: reconstruct a failed maintenance run

  • Failure modes to test before release

  • Practical lab

  • Build and test an agent trace contract

  • Evidence criteria

  • Grouped formative exercise

  • Exercise debrief

  • Key takeaway

  • Sources and further reading

  • CC-026: Reliability Engineering for Agent Failure Modes

  • Learning objectives

  • Mechanism: convert failure taxonomy into controls

  • Implementation sequence

  • The failure taxonomy

  • Detecting failure modes

  • Define service objectives around accepted outcomes

  • Spend an error budget deliberately

  • Roll back the smallest changed layer

  • Adapt conventional reliability practice to probabilistic behavior

  • Contain failures with circuit breakers and fallbacks

  • Run an agent incident response

  • Worked example: detect sycophantic screening

  • Defend the handoff boundary

  • Bound permission fan-out

  • Reliability review

  • Worked example: contain tool flailing without losing the case

  • Failure modes to test before release

  • Practical lab

  • Run an agent reliability game day

  • Evidence criteria

  • Grouped formative exercise

  • Exercise debrief

  • Key takeaway

  • Sources and further reading

  • CC-027: Cost Engineering Beyond Token Spend

  • Learning objectives

  • Mechanism: optimise the accepted outcome, not the API invoice

  • Implementation sequence

  • Measure four cost types

  • Control token and reasoning spend

  • Price wall-clock delay

  • Measure reviewer effort

  • Measure re-engagement cost

  • Include infrastructure cost

  • Build the accepted-outcome cost model

  • Monitor cost without rewarding bad outcomes

  • Find the cost-quality frontier

  • Worked example: cost a screening workload

  • Cost review

  • Measurement notes

  • Failure modes to test before release

  • Practical lab

  • Build a four-cost baseline

  • Evidence criteria

  • Grouped formative exercise

  • Exercise debrief

  • Key takeaway

  • Sources and further reading

  • Module 6 Capstone: Operate What You Ship: Observability, Reliability and Cost

  • Scenario

  • Assignment

  • Acceptance criteria

  • Rubric

  • Module 6 graded assessment

  • Module 6 graded assessment

    3 attempts allowed

  • Review guidance

  • Module 6 resources

  • Download index

  • Visuals

  • Module 6 solution guidance

  • Reviewer prompts

  • Minimum viable solution

  • Common weak submissions

  • Module 7: Scale to a Team-Owned Agent Platform

  • Outcomes

  • Lesson map

  • Completion standard

  • CC-028: From Paired Experiments to a Team-Owned Agent Platform

  • Learning objectives

  • Mechanism: make ownership the platform boundary

  • Implementation sequence

  • Pair: prove value with one accountable builder

  • Pool: establish shared team operation

  • Platform: offer agents as a supported capability

  • Choose from evidence, not fleet size alone

  • Provision through a reviewed workload contract

  • Containers are cattle, not pets

  • Scale reasoning, execution and burst capacity separately

  • Many brains, many hands

  • Dynamic workflows and burst patterns

  • Make the golden path the easiest supported route

  • Split platform and workload ownership

  • Worked example: migrate from pair to pool to platform

  • Platform readiness review

  • Design notes

  • Failure modes to test before migration

  • Practical lab

  • Design the team-owned platform transition

  • Evidence criteria

  • Grouped formative exercise

  • Exercise debrief

  • Key takeaway

  • Sources and further reading

  • CC-029: Reference Architectures, Shared Skills and Secure Orchestration

  • Learning objectives

  • Mechanism: distribute knowledge without distributing drift

  • Implementation sequence

  • Decompose the reference architecture

  • Promote one source through two deployment surfaces

  • Author skills once and detect drift

  • Centralize connector policy without centralizing credentials in agents

  • Treat cross-agent routing as a control-plane operation

  • Decide what to adopt

  • Choose build, configure or hybrid

  • Architecture review

  • Reference notes

  • Worked example: stop a poisoned handoff at the routing boundary

  • Failure modes to test before release

  • Practical lab

  • Package one workflow for two operating surfaces

  • Evidence criteria

  • Grouped formative exercise

  • Exercise debrief

  • Key takeaway

  • Sources and further reading

  • Module 7 Capstone: Scale to a Team-Owned Agent Platform

  • Scenario

  • Assignment

  • Acceptance criteria

  • Rubric

  • Module 7 graded assessment

  • Module 7 graded assessment

    3 attempts allowed

  • Review guidance

  • Module 7 resources

  • Download index

  • Visuals

  • Module 7 solution guidance

  • Reviewer prompts

  • Minimum viable solution

  • Common weak submissions

  • Module 8: Exercise Judgment: Remediate Anti-Patterns and Plan the Migration

  • Outcomes

  • Lesson map

  • Completion standard

  • CC-030: Diagnosing Anti-Patterns and Migrating Legacy Automation

  • Learning objectives

  • Mechanism: preserve deterministic strengths and isolate judgment

  • Implementation sequence

  • Diagnose a decision loop

  • Diagnose an inference loop

  • Diagnose prompt brittleness

  • Diagnose the tool trap

  • Diagnose model substitution

  • Diagnose context overload

  • Diagnose an approval bottleneck

  • Diagnose runaway cost

  • Apply the judgment test at each workflow step

  • Migrate one judgment seam under control

  • Worked example: recover from a big-bang conversion

  • Keep these operations deterministic

  • Diagnose dependency and plugin risk

  • Diagnosis review

  • Diagnostic notes

  • Failure modes to test before migration

  • Practical lab

  • Remediate a legacy automation slice

  • Evidence criteria

  • Grouped formative exercise

  • Exercise debrief

  • Key takeaway

  • Sources and further reading

  • CC-031: Architecture Judgment and a No-Hype Roadmap

  • Learning objectives

  • Mechanism: turn uncertain futures into reversible bets

  • Implementation sequence

  • Separate available capability from forecast

  • Bound plausible but unconfirmed changes

  • Reject unsupported autonomy claims

  • Build capabilities that survive roadmap changes

  • Avoid irreversible bets

  • Review architecture readiness

  • Worked example: replace a platform promise with a bounded experiment

  • Failure modes to test before release

  • Practical lab

  • Produce an evidence-led twelve-month roadmap

  • Evidence criteria

  • Grouped formative exercise

  • Exercise debrief

  • Key takeaway

  • Sources and further reading

  • Module 8 Capstone: Exercise Judgment: Remediate Anti-Patterns and Plan the Migration

  • Scenario

  • Assignment

  • Acceptance criteria

  • Rubric

  • Module 8 graded assessment

  • Module 8 graded assessment

    3 attempts allowed

  • Review guidance

  • Module 8 resources

  • Download index

  • Visuals

  • Module 8 solution guidance

  • Reviewer prompts

  • Minimum viable solution

  • Common weak submissions

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub