Building Self-Improving Agents

Architecture, Memory, Learning Systems and Practical Implementation

Introduction

Chapter 1: What Is a Self-Improving Agent

1. From Chatbots to Agents to Self-Improving Systems
2. What Counts as Improvement
3. The Seven Levers of Self-Improvement
4. Common Illusions of Progress
5. The Book’s Roadmap

Chapter 2: The Intelligence Stack: LLMs, Reasoning and Agentic Control

1. How Large Language Models Think
2. Capabilities Relevant to Agency
3. Fundamental Limitations and Failure Modes
4. Prompting as Program Execution
5. From Text Generation to Decision Making

Chapter 3: Agency and Autonomy: Control, Planning and Decision-Making

1. The Perception-Action Loop
2. Planning Paradigms: Search, Decomposition and Sampling
3. Action Spaces and Tool Use
4. Autonomy Levels and Human Oversight
5. Uncertainty, Risk and Conservative Decision Making

Chapter 4: The Memory Problem: Why Agents Forget and How to Fix It

1. Stateless Agents and the Forgetting Problem
2. A Taxonomy of Agent Memory
3. Memory as the Substrate of Improvement
4. Costs and Constraints of Memory Systems
5. Design Principles for Agent Memory

Chapter 5: Short-Term and Working Memory in Agents

1. Context Windows as Working Memory
2. Attention and Token Priority
3. Context Management Strategies
4. Sliding Windows and Context Compression
5. The Role of Short-Term Memory in Planning

Chapter 6: Long-Term Memory: Vectors, Embeddings and Retrieval

1. How Embeddings Encode Meaning
2. Vector Databases and Indexing
3. Similarity Search and Retrieval Strategies
4. Retrieval-Augmented Generation in Agents
5. Trade-Offs: Accuracy, Latency, Cost

Chapter 7: Episodic and Semantic Memory Systems

1. Episodic Memory: Storing Trajectories and Experiences
2. Semantic Memory: Facts, Concepts and Knowledge
3. Separating Events from Principles
4. Cross-Referencing Episodic and Semantic Memory
5. When to Use Each Type

Chapter 8: Procedural Memory and Skill Acquisition

1. Defining Skills in Agent Systems
2. Procedural Memory as Executable Knowledge
3. Skill Discovery and Abstraction
4. Skill Composition and Reuse
5. From Trial-and-Error to Stored Procedure

Chapter 9: Knowledge Graphs and Structured Memory

1. When Graphs Beat Vectors
2. Building and Maintaining Knowledge Graphs
3. Graph Querying in Agent Workflows
4. Hybrid Graph-Vector Memory
5. Schema Design for Agent Knowledge

Chapter 10: Memory Lifecycle: Storage, Retrieval, Update, Consolidation, Forgetting

1. Storing New Information
2. Retrieving Relevant Information Under Pressure
3. Updating Existing Beliefs
4. Memory Consolidation Across Sessions
5. The Necessity and Mechanics of Forgetting

Chapter 11: Types of Improvement: Context, Tools, Memory, Model, Code

1. Context-Based Improvement: Better Prompts and Retrieval
2. Tool-Based Improvement: New and Better Tools
3. Memory-Based Improvement: Accumulated Knowledge
4. Model-Based Improvement: Fine-Tuning and RL
5. Code and Architecture-Based Improvement

Chapter 12: Reflection-Based Learning: Self-Correction and Meta-Cognition

1. The Reflexion Pattern and Its Variants
2. Self-Critique and Error Detection
3. Post-Hoc Analysis of Failures
4. Generating Improvement Hypotheses
5. Integrating Reflection into Agent Loops

Chapter 13: Memory-Based Learning and Experience Replay

1. Experience Storage and Format Design
2. Replay Strategies and Sampling
3. Extracting Patterns from Past Trajectories
4. Combining Experience Replay with Retrieval
5. Preventing Overfitting to Past Data

Chapter 14: Automated Skill Generation and Tool Creation

1. Identifying Missing Capabilities
2. Skill Generation from Examples
3. Tool Creation and Integration
4. Testing Newly Created Skills
5. Curating the Skill Library Over Time

Chapter 15: Code Modification and Self-Evolving Systems

1. The Appeal and Danger of Self-Modification
2. Safe Code Generation Patterns
3. Automated Testing of Changes
4. Version Control and Rollback
5. Incremental vs Radical Changes

Chapter 16: Self-Generated Training Data and Supervised Fine-Tuning

1. Generating High-Quality Training Data
2. Curriculum Design for Agents
3. Data Filtering and Quality Control
4. Fine-Tuning Pipelines for Agents
5. Evaluating Fine-Tuned Models on Tasks

Chapter 17: Reinforcement Learning for Agents

1. Reward Signals and Their Design
2. RLHF for Agent Behavior
3. Online RL and Policy Optimization
4. Exploration vs Exploitation in Agents
5. Practical Constraints and Alternatives to Full RL

Chapter 18: Evolutionary and Population-Based Methods

1. Evolutionary Search for Prompts and Prompt Templates
2. Population-Based Agent Training
3. Crossover, Mutation and Selection in Agent Design
4. Large-Scale Optimization of Agent Configurations
5. When Evolutionary Methods Win

Chapter 19: Meta-Learning and Agent-Driven Experimentation

1. What Meta-Learning Means for Agents
2. Experiment Design by Agents
3. Hypothesis Testing and Validation
4. Learning Improvement Strategies Themselves
5. Hierarchical Learning Loops

Chapter 20: Single-Agent Architectures: Loops, State and Control

1. The Basic Agent Loop
2. ReAct: Reasoning Plus Acting
3. Reflexion and Self-Correction Loops
4. Self-Ask and Decomposition Patterns
5. Choosing an Architecture for Your Problem

Chapter 21: Multi-Agent Systems: Collaboration, Specialization and Competition

1. Why Multiple Agents
2. Collaboration Patterns: Handoff and Assembly
3. Specialization and Role Assignment
4. Competition and Critique Patterns
5. Communication Protocols and Alignment

Chapter 22: Hierarchical and Planner-Executor Architectures

1. Planning at Multiple Levels
2. The Planner-Executor Divide
3. Managing Subgoals and Dependencies
4. Hierarchical Task Networks
5. Delegation and Oversight

Chapter 23: Critic-Evaluator Systems and Learning Loops

1. The Role of a Critic
2. Designing Effective Evaluators
3. Automated vs Human Evaluation
4. Integrating Feedback into Improvement
5. Avoiding Evaluation Gaming

Chapter 24: Persistent Autonomous Agents

1. Long-Horizon Operation
2. State Persistence Across Sessions
3. Resource Management Over Time
4. Drift Detection and Correction
5. The Promise and Risk of Persistent Agents

Chapter 25: End-to-End Implementation: A Self-Improving Agent System

1. Technology Stack and Project Structure
2. Requirements and Dependencies
3. Core Data Models
4. Memory System Integration
5. Core Agent Loop Implementation
6. Experience Storage and Retrieval
7. Reflection and Self-Evaluation
8. Improvement Generation and Validation
9. Configuration, Testing and Deployment
10. Running and Extending the System

Chapter 26: Evaluation: Measuring Genuine Improvement

1. What to Measure and Why
2. Benchmark Design for Self-Improving Agents
3. A/B Testing and Controlled Experiments
4. Regression Detection
5. Separating Signal from Noise

Chapter 27: Failure Modes and Risks

1. Hallucinated and Corrupted Memories
2. Reward Hacking and Goodhart’s Law
3. Compounding Errors and Drift
4. Overfitting and Catastrophic Forgetting
5. Security Vulnerabilities in Self-Modifying Agents

Chapter 28: Safeguards and Reliability Engineering

1. Sandboxing and Permission Boundaries
2. Human Approval Gates
3. Version Control and Immutable Audit Logs
4. Reproducible Experiments
5. Rollback Mechanisms and Kill Switches

Chapter 29: Deployment, Monitoring and Production Systems

1. Observability and Logging
2. Performance Monitoring and Budgets
3. Scaling and Resource Management
4. Incident Response for Autonomous Systems
5. Long-Term Maintenance

Conclusion

References