Engineering Memory for AI Agents

From First Principles to Production Systems

Introduction

1. The Problem with “Just Use a Larger Context Window”
2. What Agent Memory Actually Is
3. How This Book Is Organized
4. Prerequisites

Chapter 1: Why Agents Need Memory

1. The Context Window Is Not Memory
2. What Happens When Agents Forget
3. Defining Agent Memory vs Related Concepts
4. The Cost of Forgetting in Real Systems

Chapter 2: A Taxonomy of Agent Memory

1. Working Memory and Short-Term State
2. Episodic Memory for Interaction Histories
3. Semantic Memory for Facts and Knowledge
4. Procedural Memory for Skills and Patterns
5. Entity, User, Task and Environmental Memory
6. Shared Multi-Agent Memory

Chapter 3: The Memory Lifecycle from Capture to Retrieval

1. Observing and Capturing Candidate Memories
2. Deciding What Is Worth Remembering
3. Extracting Facts, Entities and Relations
4. Structuring and Enriching Raw Observations
5. From Capture to Storage: The Memory Pipeline

Chapter 4: Representing Memories: Schemas and Data Structures

1. Designing a Memory Schema
2. Generating Embeddings for Semantic Search
3. Importance, Confidence and Decay Scores
4. Deduplication and Consolidation Strategies
5. Temporal Versioning and History Tracking
6. Forgetting Policies and Expiration

Chapter 5: Storage Foundations from Files to Databases

1. In-Memory Structures and File-Based Storage
2. SQLite with Full-Text Search (FTS5)
3. PostgreSQL for Structured Memory at Scale
4. Vector Indexes and Approximate Nearest Neighbor
5. Document Stores, Key-Value Stores, and Object Storage
6. Graph Databases for Relational Memory

Chapter 6: Retrieval Systems for Finding What Matters

1. Lexical Retrieval with BM25
2. Vector Similarity Search Internals
3. Hybrid Retrieval: Combining Lexical and Semantic
4. Metadata Filtering and Constrained Search
5. Reciprocal Rank Fusion and Multi-Signal Ranking
6. Rerankers and Cross-Encoders
7. Temporal and Graph-Based Retrieval

Chapter 7: Context Engineering for Injecting Memories into Prompts

1. Token Budgets and Context Constraints
2. Relevance vs Recency in Memory Selection
3. Dynamic Context Assembly Strategies
4. Hierarchical and Recursive Summarization
5. Memory Scopes, Namespaces, and Isolation
6. Prompt Construction with Retrieved Memories

Chapter 8: Memory Architectures for Agent Types

1. Memory for Tool-Using Agents
2. Coding Agents and Repository-Aware Memory
3. Research Agents and Long-Horizon Reasoning
4. Personal Assistants and User-Centric Memory
5. Multi-Agent Systems and Shared Memory
6. Autonomous Workflows, Plans, and Checkpoints

Chapter 9: Frameworks, Integrations, and Interoperability

1. Vendor-Independent Memory Architecture
2. Integrating with Model APIs and Tool Calling
3. The Model Context Protocol (MCP) for Memory Tools
4. Plugging into Agent Frameworks Without Lock-In
5. Designing a Memory Service API

Chapter 10: Correctness, Reliability, and Consistency

1. Stale Memories and Knowledge Drift
2. Contradictions and Conflicting Information
3. Provenance, Attribution, and Trust
4. Deterministic vs Probabilistic Retrieval Guarantees
5. Concurrency, Race Conditions, and Consistency
6. Backups, Migrations, and Disaster Recovery

Chapter 11: Security, Privacy, and Access Control

1. Access Control and Tenant Isolation
2. Encryption at Rest and in Transit
3. Handling Secrets and Sensitive Data
4. Prompt Injection Through Stored Memories
5. Memory Poisoning and Malicious Content
6. Deletion, Retention, and Auditability

Chapter 12: Evaluating Agent Memory

1. Retrieval Metrics: Precision, Recall, and Ranking
2. Task-Success Measurement with Memory
3. Long-Horizon Benchmarks and Stability Tests
4. Cost, Latency, and Efficiency Metrics
5. Ablation Testing and Causal Attribution
6. Building Repeatable Evaluation Pipelines

Chapter 13: Observability, Debugging, and Diagnostics

1. Memory Traces and Retrieval Logs
2. Ranking Explanations and Decision Auditing
3. Token Accounting and Context Visibility
4. Interactive Memory Inspection Tools
5. Diagnosing Common Failure Modes

Chapter 14: Performance, Cost, and Scale Engineering

1. Embedding Costs and Optimization Strategies
2. Query Latency and Indexing Performance
3. Storage Growth Modeling and Cost Projections
4. Caching, Batching, and Throughput
5. Partitioning, Sharding, and Replication
6. Token Cost Engineering

Chapter 15: Building a Production Memory System via a Staged Roadmap

1. Stage 1: Minimal Memory with Conversation History
2. Stage 2: Persistent Structured Memory in SQLite
3. Stage 3: Lexical and Semantic Retrieval with Hybrid Ranking
4. Stage 4: Consolidation, Forgetting, and Temporal Reasoning
5. Stage 5: Provenance, Evaluation, Observability, and Security
6. Stage 6: Scalable Multi-Agent Production Architecture

Conclusion: The Engineering Discipline of Agent Memory

1. What We Know
2. What Remains Uncertain
3. When Simple Is Better
4. The Path Forward

References