Building AI Agents and Agent Harnesses from Scratch

Internals, Architecture, Algorithms and Production Implementation

Introduction

1. What You Are Really Building
2. Why You Should Not Use a Black Box
3. How This Book Is Structured
4. What You Need to Know
5. What This Book Is Not
6. A Note on Examples and Code
7. The Central Thesis

Chapter 1: The Agent Mental Model

1. What an agent is not: dispelling the illusion of autonomous models
2. The harness as execution environment: orchestrating the loop
3. Responsibility boundaries: model, inference runtime, harness, tools
4. A concrete mental model with a layered diagram
5. Why this architecture is necessary: the gap between generation and action

Chapter 2: How Large Language Models Actually Work

1. Tokenization: how text becomes numbers the model sees
2. The context window: what the model “remembers” and its limits
3. Inference: forward passes, attention, sampling strategies
4. Temperature, top-p, top-k: controlling randomness
5. Structured outputs and why they are difficult
6. The model as a stateless function: f(context) → tokens

Chapter 3: The Model API as a Contract

1. Chat completions API: the universal interface
2. Message roles: system, user, assistant and tool messages
3. System instructions and their actual effect on behavior
4. The OpenAI-compatible API standard and its spread
5. Provider-specific extensions: function calling, tool choice, reasoning models
6. Latency, rate limits and operational characteristics

Chapter 4: From Prompt to Action: The Agent Execution Cycle

1. A complete execution trace of a simple agent request
2. Step one: receiving the request and initializing state
3. Step two: assembling context from history, tools and prompt
4. Step three: invoking the model and receiving a response
5. Step four: parsing the model response for actions
6. Step five: executing tools and collecting observations
7. Step six: feeding results back and deciding whether to continue
8. Step seven: the model generates the final answer
9. Termination: when the agent decides it is done
10. What happens inside the model vs what happens inside the harness

Chapter 5: Tool Calling - From Model Output to Real Execution

1. Tool definitions: schemas, descriptions and parameters
2. How models learn to generate tool calls
3. The tool call response format: structure and variants
4. Parsing model outputs: JSON, function calls and structured formats
5. Argument validation and schema enforcement
6. The dispatch mechanism: routing a tool name to handler code
7. Returning observations to the model: formatting and context cost

Chapter 6: The Prompt and Context Assembly Problem

1. The components of agent context: system prompt, history, tools, state
2. Prompt templates and dynamic assembly
3. Token budgeting: fitting everything into the window
4. Truncation strategies: what to keep and what to drop
5. Cost of tool schemas and history growth
6. When the context window becomes a bottleneck

Chapter 7: State, Memory and Conversation Management

1. Ephemeral state: the active execution context
2. Conversation history: storing and serializing message sequences
3. Message compression and summary strategies
4. Working memory: scratchpad and intermediate state
5. Long-term memory: vector stores and retrieval
6. Memory isolation: per-session vs shared knowledge

Chapter 8: Project Setup and First Contact

1. Technology choices: why Python, why an OpenAI-compatible interface
2. Project structure: organization for a growing codebase
3. Dependency management: pinning versions
4. Configuration: environment variables and config objects
5. The minimal model invocation: first request and response
6. Error handling for model API calls
7. Running and testing the initial setup

Chapter 9: The Core Message and State Types

1. Message types: SystemMessage, UserMessage, AssistantMessage, ToolMessage
2. Tool definitions: schema, parameters, descriptions as data structures
3. Agent state: history, active tools, configuration, limits
4. Serialization: JSON, persistence and round-tripping
5. Type safety: using dataclasses and protocols
6. Versioning: handling schema changes over time

Chapter 10: The Tool Registry and Dispatch System

1. Tool registration: decorating or registering functions as tools
2. Schema generation: extracting JSON Schema from Python types
3. Tool description language: what makes a tool callable by a model
4. The registry data structure and lookup
5. Dispatch: matching tool names and routing to handlers
6. Result formatting: converting Python values to model-readable strings

Chapter 11: The Agent Execution Loop

1. Loop design: the core while loop with exit conditions
2. Iteration limits: preventing infinite execution
3. Token budgets: tracking and enforcing limits
4. Maximum tool calls per turn and overall
5. Response parsing: handling text-only, tool-call and mixed responses
6. Error handling within the loop
7. Graceful termination and timeout handling

Chapter 12: Streaming, Concurrency and Async Operations

1. Streaming model responses: chunked data and incremental parsing
2. Building streaming agents: yielding tokens and events as they arrive
3. Async tool execution: why some tools need await
4. Parallel tool calls: executing multiple tools concurrently
5. Race conditions and ordering in parallel execution
6. Cancellation: interrupting a running agent

Chapter 13: Persistence, Checkpoints and Durability

1. Why persistence matters: crash recovery and resumption
2. State serialization: what to save and when
3. Checkpointing strategy: periodic vs event-driven
4. Database choices: SQLite, Postgres, key-value stores
5. Resuming from checkpoints: restoring and continuing execution

Chapter 14: Error Handling, Retries and Fault Tolerance

1. Failure taxonomy: network errors, API errors, model errors, tool errors
2. Retry policies: exponential backoff, jitter and limits
3. Idempotency: safe retries for tool execution
4. Handling malformed model outputs
5. Fallback strategies and circuit breakers
6. Timeout handling at every layer

Chapter 15: Observability, Logging and Tracing

1. Logging levels and what to log at each level
2. Structured logging: JSON logs for machine readability
3. Event streaming: execution events as a first-class concern
4. Distributed tracing: tracking requests across services
5. Metrics: latency, token usage, error rates
6. Debugging real agent runs: reconstructing what happened

Chapter 16: Security, Sandboxing and Trust Boundaries

1. The trust model: what is trusted and what is not
2. Prompt injection: how it works and defense strategies
3. Tool authorization: never trusting the model to respect permissions
4. Input validation and sanitization at every boundary
5. Sandboxing tool execution: containers, timeouts, resource limits
6. Secrets management and credential safety
7. Rate limiting and abuse prevention

Chapter 17: Performance, Cost and Resource Optimization

1. Latency profiling: where time is spent in the agent loop
2. Token cost optimization: concise prompts and selective context
3. Model selection strategies: smaller models for simpler tasks
4. Caching strategies: prompt caching and response caching
5. Batch processing: running multiple agents efficiently

Chapter 18: Planning, Decomposition and Control Strategies

1. Simple reactive agents vs planning agents
2. Task decomposition: breaking problems into steps
3. Plan-and-execute: generating and following a plan
4. Self-reflection and self-correction patterns
5. Tree-of-thoughts and graph search in agents

Chapter 19: Hierarchical Agents and Multi-Agent Systems

1. Why multiple agents: specialization and isolation
2. Hierarchical architectures: manager and worker agents
3. Agent communication: protocols for inter-agent coordination
4. Shared tools and shared context
5. Routing: choosing which agent handles which task
6. Emergent complexity in multi-agent systems

Chapter 20: Retrieval-Augmented Generation and Knowledge Access

1. Why RAG: extending agents beyond training data
2. Chunking, embedding and indexing strategies
3. Retrieval as a tool vs embedded context
4. Reranking and relevance filtering
5. Grounded generation: constraining outputs to retrieved content
6. Evaluating retrieval quality

Chapter 21: Long-Running Workflows and Human-in-the-Loop

1. Long-running agents vs request-response agents
2. Workflow orchestration: state machines and DAGs
3. Human approval gates: pausing and resuming with human input
4. Webhooks and callbacks for asynchronous workflows
5. Timeout and staleness handling
6. Idempotent workflow execution

Chapter 22: Evaluating Agents - Correctness, Reliability and Safety

1. What to measure: correctness, reliability, latency, cost, safety
2. Example-based evaluation: test suites for agent behaviors
3. Automated evaluation with model-as-judge
4. Regression testing: catching breaking changes
5. Safety evaluation: stress testing for injection and abuse
6. Observability-driven improvement

Chapter 23: From Minimal Loop to Production Harness

1. The minimal viable agent: what you can start with
2. Capability tiers: what to add and when
3. Architecture review: the complete system
4. Common pitfalls and how to avoid them
5. Making trade-offs: simplicity vs capability
6. Maintenance and evolution of agent systems

Chapter 24: Open Problems and the Future of Agent Harnesses

1. Unsolved reliability problems
2. The role of improved models vs better harnesses
3. Standardization efforts and protocols
4. Agent-to-agent communication and ecosystems
5. Integration with operating systems and infrastructure
6. Speculative directions and what to watch

Conclusion

References