Agent Specification Engineering

Agent Specification Engineering

1. Building Reliable, Testable, and Governable AI Systems with DSLs, Contracts, and Executable Specifications

Copyright

1. Code License

Disclaimer

About the Author

Preface

How to Use This Book

Terminology and Notation

The Missing Engineering Layer Between Intent and Execution

1. The Production Problem Is Larger Than Answer Quality
2. Prompting and Specification Solve Different Problems
3. Goals Are Not Instructions, and Autonomy Is Not Authority
4. A First AgentSpec
5. Executable Does Not Mean Fully Formal
6. The Reader Promise

Part I — Why Agents Need Specifications

Chapter 1 — The Agent Reliability Problem

1. The Demonstration Hides the State Space
2. Eight Reliability Failures Behind a Fluent Answer
3. Why a Better Prompt Cannot Close These Gaps
4. Unsafe Design: The Prompt as Control Plane
5. Corrected Design: Proposals Inside an Enforcement Shell
6. Failure Semantics Must Be Part of the Product
7. Security and Privacy Are Execution Properties
8. Reliability and Operational Implications
9. When Not to Build an Agent
10. Design Review: A Fluent but Unreliable Support Agent
11. Production Checklist
12. Key Takeaways
13. Exercises

Chapter 2 — From Prompts to Behavioral Contracts

1. A Ladder of Explicitness
2. Design by Contract, Adapted Carefully
3. Preconditions Are Not Suggestions to Prepare Better
4. Postconditions Describe Observable Guarantees
5. Invariants Protect the Execution Across Steps
6. An Insufficient Structured Specification
7. A Contract-Oriented Refactoring
8. What Belongs in Prose, Data, Code, or Policy
9. Guardrails Are Not More Instructions
10. Security, Reliability, and Operations
11. Trade-offs and When Not to Formalize Further
12. Design Review: Refactoring an Operations Prompt
13. Production Checklist
14. Key Takeaways
15. Exercises

Chapter 3 — Anatomy of an Agent Specification

1. AgentSpec’s Design Boundary
2. The Minimal Specification
3. Metadata and Portability Make Context Reproducible
4. Vocabulary Prevents Domain Drift
5. Inputs and Outputs Define Ports, Not Merely Formats
6. Objectives Separate Direction from Termination
7. Capabilities and Tools Separate Ability from Adapters
8. The Contract Section Defines Cross-Cutting Obligations
9. State Turns a Loop into a Workflow
10. Policies Express Contextual Authority
11. Memory, Context, and Data Are Three Policies
12. Budgets and Resilience Define the Outer Failure Envelope
13. Observability and Audit Serve Different Consumers
14. Evaluation Makes Release Expectations Executable
15. The Insufficient Specification Revisited
16. Implementation and Deployment Considerations
17. When AgentSpec Is Not the Right Representation
18. Design Review: Finding the Missing Section
19. Production Checklist
20. Key Takeaways
21. Exercises

Part II — Designing Agent Contracts

Chapter 4 — Goals, Outcomes, and Completion Conditions

1. A Vocabulary of Ending
2. Goals Need Bounds
3. Output Is Not Outcome
4. Acceptance Criteria Belong to Different Owners
5. Completion Must Be a Runtime Decision
6. Terminal States Are Semantically Different
7. Business Success Is Often Delayed
8. Model Confidence Is One Signal
9. An Unsafe Completion Design
10. Cancellation and Changing Intent
11. Security, Reliability, and Operational Implications
12. When Not to Use a Complex Outcome Model
13. Design Review: When “Resolved” Is Three Different Things
14. Production Checklist
15. Key Takeaways
16. Exercises

Chapter 5 — Tool Contracts and Side Effects

1. A Tool Is More Than a Function Signature
2. Typed Inputs and Outputs
3. Read, Reversible Write, Irreversible Write
4. Preconditions and Postconditions Around Effects
5. Timeouts Create Knowledge Problems
6. Designing Idempotency Keys
7. Retry Is a Classified Decision
8. Compensation Is Forward Recovery
9. Human Approval Belongs Before the Effect
10. Tool-Result Provenance
11. An Unsafe and Corrected Tool Design
12. Security, Privacy, Reliability, and Operations
13. When Not to Expose a Tool
14. Design Review: The Timeout After Commit
15. Production Checklist
16. Key Takeaways
17. Exercises

Chapter 6 — State, Memory, and Context

1. Four Information Planes
2. Workflow State Is Not a Transcript
3. Domain State Has an Owner
4. Context Is a Purpose-Built Projection
5. Memory Needs a Retention Contract
6. Staleness Is a Semantic Property
7. Conflicting Context Must Remain Visible
8. Context Minimization and Privacy
9. An Unsafe and Corrected Design
10. Reliability and Operational Implications
11. When Not to Use Persistent Memory
12. Design Review: The Convenient Summary That Became Truth
13. Production Checklist
14. Key Takeaways
15. Exercises

Chapter 7 — Decisions and State Transitions

1. The Model Proposes; the State Machine Commits
2. States Should Represent Operational Meaning
3. Valid and Invalid Transitions
4. Guards Are Current Predicates
5. Transition Effects Need Their Own Safety
6. Terminal, Suspended, and Unknown States
7. Loops Need Progress Measures
8. Temporal Rules
9. Recovery and Versioning
10. An Unsafe and Corrected Workflow
11. Security and Operational Implications
12. When Not to Use a Rich State Machine
13. Design Review: Two Workers, One Approval
14. Production Checklist
15. Key Takeaways
16. Exercises

Chapter 8 — Policies, Permissions, and Authority

1. Authority Is a Tuple, Not a Boolean
2. Least Privilege Starts With Architecture
3. Capabilities Are Delegable Units of Authority
4. Roles and Attributes Answer Different Questions
5. Deny by Default and Explicit Prohibition
6. Policy as Code
7. Tenant and Regional Boundaries
8. Approval Thresholds and Separation of Duties
9. Prompt Injection as an Authorization Problem
10. Confused Deputies
11. An Unsafe and Corrected Engineering Agent
12. Reliability and Operational Implications
13. When Not to Delegate Authority
14. Design Review: The Confused Deputy in a Repository Agent
15. Production Checklist
16. Key Takeaways
17. Exercises

Part III — Execution Architecture

Chapter 9 — Compiling Specifications into Workflows

1. Compilation as a Pipeline
2. Parsing and Normalization
3. Structural Validation
4. Semantic Validation
5. Binding to Deployment Reality
6. From State Model to Workflow Graph
7. Planning Is Constrained Graph Construction
8. Model-Facing Projections
9. Generated Runtime Artifacts
10. Runtime Execution
11. An Unsafe and Corrected Compiler
12. Security and Supply-Chain Considerations
13. Operational Implications
14. When Not to Generate
15. Design Review: The Specification Passed but the Deployment Drifted
16. Production Checklist
17. Key Takeaways
18. Exercises

Chapter 10 — Deterministic and Probabilistic Boundaries

1. Five Responsibility Zones
2. What the Model Should Interpret
3. What Deterministic Code Should Validate
4. What Policy Engines Should Authorize
5. What Transaction Services Should Execute
6. What Humans Should Approve
7. The Deterministic Shell
8. Constrained Decoding and Schema Enforcement
9. Confidence Thresholds
10. False Determinism
11. An Unsafe and Corrected Decision Path
12. Latency and Cost Trade-offs
13. Security and Operational Implications
14. When Not to Use a Model
15. Design Review: JSON That Is Valid and Wrong
16. Production Checklist
17. Key Takeaways
18. Exercises

Chapter 11 — Orchestration, Delegation, and Multi-Agent Systems

1. Multi-Agent Is an Organizational Choice
2. Supervisor and Specialists
3. Delegation Contracts
4. Task Ownership
5. Authority Propagation
6. Shared State Without Shared Confusion
7. Handoff Schemas
8. Result Validation and Trust
9. Coordination Versus Decomposition
10. Circular Delegation and Termination
11. Unsafe and Corrected Multi-Agent Design
12. Security and Privacy
13. Reliability and Operations
14. When Not to Use Multiple Agents
15. Design Review: When More Agents Create Less Accountability
16. Production Checklist
17. Key Takeaways
18. Exercises

Chapter 12 — Failure Semantics and Recovery

1. Failure Is a Contract Outcome
2. Transient and Permanent Failures
3. Invalid Model Output
4. Tool Timeout and Unknown Outcome
5. Partial Execution
6. Duplicate and Out-of-Order Results
7. Dependency Failure
8. Approval Timeout and Denial
9. Conflicting Decisions
10. Backoff and Jitter
11. Dead-Letter Handling
12. Compensation and Escalation
13. Retry Versus Safe Replay
14. Failure Budgets
15. Unsafe and Corrected Recovery
16. Security and Privacy
17. Operational Implications
18. When Not to Recover Automatically
19. Design Review: Recovery Without Wishful Rollback
20. Production Checklist
21. Key Takeaways
22. Exercises

Part IV — Verification and Quality

Chapter 13 — Testing Agent Specifications

1. A Layered Test Strategy
2. Schema Tests
3. Parser and Normalization Tests
4. Semantic Tests
5. Contract Tests for Tools
6. State-Transition Tests
7. Invariant Tests
8. Property-Based Testing
9. Model-Based Testing
10. Simulation and Failure Injection
11. Adversarial Tests
12. Golden Datasets and Regression Suites
13. Mutation Testing
14. Test Doubles and Deterministic Models
15. Stochastic Test Strategy
16. Security and Operations
17. When Not to Use an End-to-End Model Test
18. Design Review: A Test Suite That Passed the Wrong System
19. Production Checklist
20. Key Takeaways
21. Exercises

Chapter 14 — Evaluating Behavior, Not Just Answers

1. Evaluation Begins With a Behavioral Record
2. Separate Hard Constraints From Quality
3. Response Quality
4. Task Completion
5. Tool Selection and Argument Correctness
6. Policy Compliance and Execution Order
7. Escalation Quality
8. Evidence Use and Source Attribution
9. Latency, Cost, and Reliability
10. Behavioral Scorecards
11. Offline Evaluation
12. Online Evaluation
13. Evaluation Leakage
14. Evaluator Disagreement
15. Unsafe and Corrected Evaluation
16. Security and Privacy
17. When Not to Use a Single Metric
18. Design Review: The 92 Percent Agent
19. Production Checklist
20. Key Takeaways
21. Exercises

Chapter 15 — Guardrails and Runtime Enforcement

1. A Guardrail Must Control a Boundary
2. Input Validation
3. Output Validation
4. Domain Validation
5. Policy and Authorization
6. Rate Limits and Budgets
7. Sandboxing
8. Content Controls
9. Data-Loss Prevention
10. Approval Gates
11. Kill Switches and Capability Rollback
12. Layered Guardrails
13. Unsafe and Corrected Engineering Guardrails
14. Guardrail Composition and Failure
15. Security and Privacy
16. When Not to Add Another Guardrail
17. Design Review: The Guardrail That Lived in the Prompt
18. Production Checklist
19. Key Takeaways
20. Exercises

Chapter 16 — Observability, Auditability, and Replay

1. Three Evidence Products
2. Traces and Spans
3. Execution Timelines
4. Decision Records
5. Version Identity
6. Tool Calls and Policy Decisions
7. Provenance
8. Token and Cost Accounting
9. Audit Logs
10. Redaction
11. Correlation Identifiers
12. Safe Replay
13. Forensic Investigation
14. Unsafe and Corrected Observability
15. Privacy Implications
16. When Not to Store a Transcript
17. Design Review: Reconstructing a Duplicate Effect
18. Production Checklist
19. Key Takeaways
20. Exercises

Part V — Production Patterns

Chapter 17 — Customer-Support Triage Agent

1. Scope and Architecture
2. AgentSpec Boundary
3. Data Model
4. State Machine
5. Tool Contracts
6. Policies and Privacy
7. Classification and Deterministic Rules
8. Testing Strategy
9. Walkthrough: The Suspicious-Login Ticket
10. Decision Table
11. Data-Threat Review
12. Observability and Audit
13. Failure Analysis
14. Unsafe Design
15. Corrected Design and Trade-offs
16. When Not to Use This Agent
17. Design Review: Mixed Intent Under Degraded Dependencies
18. Production Checklist
19. Key Takeaways
20. Exercises

Chapter 18 — Software-Engineering Agent

1. Scope and Architecture
2. AgentSpec Boundary
3. Repository Permissions
4. Sandboxing and Command Contracts
5. Planning Contract
6. Patch and Artifact Provenance
7. Test Gates
8. Pull-Request Preparation and Human Review
9. Walkthrough: Protected Workflow Drift
10. State and Failure Paths
11. Evaluation and Tests
12. Observability
13. Trade-offs
14. When Not to Use This Agent
15. Design Review: A Correct Patch from an Unsafe Process
16. Production Checklist
17. Key Takeaways
18. Exercises

Chapter 19 — Financial-Operations Agent

1. Scope and Architecture
2. AgentSpec Boundary
3. Authoritative Data and Immutable Evidence
4. Threshold Policy
5. Separation of Duties
6. Review Package
7. State and Human Authorization
8. Idempotency and Duplicate Containment
9. Compensation and Reversal
10. Data Privacy and Explainability
11. Walkthrough: Threshold Drift
12. Tests and Evaluation
13. Observability and Audit
14. Failure Containment
15. Trade-offs and When Not to Use
16. Design Review: The Threshold at the Currency Cutoff
17. Production Checklist
18. Key Takeaways
19. Exercises

Chapter 20 — Multi-Agent Incident Response

1. Roles and Authority
2. Architecture
3. Supervisor AgentSpec
4. Immutable Incident Timeline
5. Evidence Collection
6. Conflicting Hypotheses
7. Time Bounds and Budgets
8. Blast-Radius Control
9. Remediation State Machine
10. Communications Contract
11. Walkthrough: The Wrong-Region Hypothesis
12. Failure and Recovery
13. Testing and Evaluation
14. Observability and Post-Incident Replay
15. Trade-offs and When Not to Use
16. Design Review: Approval Invalidated by Late Telemetry
17. Production Checklist
18. Key Takeaways
19. Exercises

Chapter 21 — Portability, Evolution, and Governance

1. Model Portability
2. Framework Portability
3. Specification Versioning
4. Schema Evolution
5. Migration
6. Deprecation
7. Governance as an Engineering System
8. Review Processes
9. Agent Catalogs
10. Organizational Maturity
11. Production-Readiness Gates
12. Unsafe and Corrected Evolution
13. Security and Operational Implications
14. When Not to Pursue Portability
15. Design Review: A Portable Specification on an Incomplete Framework
16. Production Checklist
17. Key Takeaways
18. Exercises

From Agent Demos to Engineered Systems

1. The Discipline of Agent Specification Engineering
2. The Central Separations
3. A Practical Adoption Sequence
4. A Ninety-Day Adoption Plan
5. What AgentSpec Contributes
6. The Standard for “Reliable”
7. The Engineering Model to Carry Forward

Appendix A — AgentSpec 0.1 Language Reference

1. Document and Serialization
2. Conformance Levels
3. Top-Level Inventory
4. metadata
5. agent
6. purpose and vocabulary
7. Data Ports: inputs and outputs
8. objectives
9. capabilities
10. tools
11. Predicates
12. contract
13. state
14. policies
15. memory
16. context
17. data
18. budgets
19. resilience
20. observability
21. audit
22. evaluation
23. portability
24. Validation Order
25. Minimum Runtime Events
26. Compatibility and Migration

Appendix B — AgentSpec JSON Schema

1. Dialect and Identity
2. Closed Top-Level Structure
3. Reusable Definitions
4. Required Versus Optional Fields
5. Enumerations
6. Bounds
7. Arrays and Uniqueness
8. Embedded Port and Tool Schemas
9. Schema Validation in the Reference Engine
10. Versioning the Schema
11. Using the Schema in CI
12. Complete AgentSpec 0.1 Schema

Appendix C — Threat-Modeling Checklist

1. 1. System and Purpose
2. 2. Assets
3. 3. Entry Points and Trust Boundaries
4. 4. Identity and Delegation
5. 5. Capability and Tool Authority
6. 6. Prompt Injection and Untrusted Content
7. 7. Data Privacy and Isolation
8. 8. Context and Memory Integrity
9. 9. Model and Provider Boundary
10. 10. State and Workflow Integrity
11. 11. Side Effects and Distributed Failure
12. 12. Sandbox and Execution Environment
13. 13. Human Approval and Social Engineering
14. 14. Output and External Communication
15. 15. Observability, Audit, and Replay
16. 16. Supply Chain and Control Plane
17. 17. Availability and Abuse
18. 18. Evaluation and Detection
19. 19. Residual Risk and Decision
20. Threat-Model Review Record

Appendix D — Production-Readiness Checklist

1. Release Header
2. Gate 1 — Purpose, Ownership, and Scope
3. Gate 2 — Specification and Build Integrity
4. Gate 3 — Identity, Authorization, and Secrets
5. Gate 4 — Data Governance and Privacy
6. Gate 5 — Tooling and Side-Effect Safety
7. Gate 6 — State, Concurrency, and Recovery
8. Gate 7 — Model and Context Behavior
9. Gate 8 — Sandbox, Network, and Supply Chain
10. Gate 9 — Verification and Evaluation
11. Gate 10 — Observability and Audit
12. Gate 11 — Operations and Incident Response
13. Gate 12 — Rollout and Change Control
14. Release Decision
15. Fast Red-Flag Review

Appendix E — Evaluation Scorecards

1. Scorecard Header
2. Universal Hard Gates
3. Behavioral Rating Scale
4. Core Behavioral Dimensions
5. Weighting Template
6. Risk Slices
7. Customer-Support Triage Scorecard
8. Software-Engineering Agent Scorecard
9. Financial-Operations Agent Scorecard
10. Multi-Agent Incident Scorecard
11. Online Evaluation and Stop Conditions
12. Adjudication Protocol
13. Completed Scorecard Record

Appendix F — Exercise Solutions and Discussion Notes

1. Chapter 1 — The Agent Reliability Problem
2. Chapter 2 — From Prompts to Behavioral Contracts
3. Chapter 3 — Anatomy of an Agent Specification
4. Chapter 4 — Goals, Outcomes, and Completion Conditions
5. Chapter 5 — Tool Contracts and Side Effects
6. Chapter 6 — State, Memory, and Context
7. Chapter 7 — Decisions and State Transitions
8. Chapter 8 — Policies, Permissions, and Authority
9. Chapter 9 — Compiling Specifications into Workflows
10. Chapter 10 — Deterministic and Probabilistic Boundaries
11. Chapter 11 — Orchestration, Delegation, and Multi-Agent Systems
12. Chapter 12 — Failure Semantics and Recovery
13. Chapter 13 — Testing Agent Specifications
14. Chapter 14 — Evaluating Behavior, Not Just Answers
15. Chapter 15 — Guardrails and Runtime Enforcement
16. Chapter 16 — Observability, Auditability, and Replay
17. Chapter 17 — Customer-Support Triage Agent
18. Chapter 18 — Software-Engineering Agent
19. Chapter 19 — Financial-Operations Agent
20. Chapter 20 — Multi-Agent Incident Response
21. Chapter 21 — Portability, Evolution, and Governance

Appendix G — Glossary

Appendix H — Bibliography and Further Reading

1. Contracts, Formal Reasoning, and State
2. Data Formats, APIs, and Events
3. Security, Privacy, and Risk
4. Observability, Reliability, and Supply Chain
5. Language Models, Reasoning, and Tools
6. Suggested Reading Paths
7. Source Maintenance Note