Building AI Agents with Ollama

Design, Deploy, and Scale Reliable Local AI Systems

Introduction: The Case for Local Agents

1. Why Local Matters Now
2. What Is an AI Agent
3. The Reliability Gap
4. How This Book Is Structured
5. Key Takeaways

Chapter 1: Foundations of Local LLMs and Ollama

1. How Local Inference Works
2. Installing and Configuring Ollama
3. Understanding the Ollama API
4. Model Formats and GGUF
5. First Agent: A Minimal Example
6. Key Takeaways

Chapter 2: Choosing and Managing Models

1. The Local Model Landscape
2. Performance vs Size Tradeoffs
3. Quantization Explained
4. Multi-Model Strategies
5. Key Takeaways

Chapter 3: Prompt Engineering for Agents

1. From Prompts to Specifications
2. Structured Prompt Design Patterns
3. System Prompts and Role Definitions
4. Few-Shot and Example-Based Prompting
5. Testing and Iterating Prompts
6. Key Takeaways

Chapter 4: Structured Outputs and Tool Calling

1. JSON Mode and Schema Enforcement
2. Pydantic Models for Type Safety
3. Function Calling Patterns
4. Error Recovery for Bad Outputs
5. Building a Tool Registry
6. Browser Automation Tools
7. Key Takeaways

Chapter 5: Memory Systems and Context Management

1. Chapter Dependencies and Project Structure
2. The Memory Problem
3. Short-Term Conversation Memory
4. Long-Term Memory Architectures
5. Context Window Optimization
6. Implementing a Memory Manager
7. Key Takeaways

Chapter 6: Retrieval-Augmented Generation and Vector Search

1. Chapter Dependencies and Project Structure
2. Why RAG Is Essential for Agents
3. Embeddings and Vector Spaces
4. Vector Database Options
5. Document Ingestion Pipelines
6. Hybrid Search and Re-Ranking
7. Key Takeaways

Chapter 7: Planning, Reasoning, and Complex Tasks

1. Beyond Single-Turn Responses
2. Chain of Thought and Reasoning
3. The ReAct Pattern
4. Task Decomposition Strategies
5. Key Takeaways

Chapter 8: Multi-Agent Architectures

1. Why Multiple Agents
2. Specialization and Roles
3. Communication Patterns
4. Orchestration Strategies
5. Building an Agent Team
6. Key Takeaways

Chapter 9: Workflows, State Machines, and Orchestration

1. Agents as Workflow Nodes
2. State Machine Design
3. Async Processing and Concurrency
4. Retries, Timeouts, and Circuit Breakers
5. Key Takeaways

Chapter 10: Testing, Debugging, and Observability

1. Chapter Dependencies and Project Structure
2. The Testing Challenge
3. Unit Testing Agents and Tools
4. Integration and End-to-End Tests
5. Logging and Tracing
6. Monitoring and Alerting
7. Key Takeaways

Chapter 11: Performance, Resources, and Optimization

1. Understanding Resource Consumption
2. GPU Acceleration and Configuration
3. Caching Strategies
4. Batch Processing and Throughput
5. Scaling Local Inference
6. Key Takeaways

Chapter 12: Security, Privacy, and Sandboxing

1. The Threat Model for Local Agents
2. Prompt Injection Attacks
3. Tool Sandboxing
4. Data Privacy and Compliance
5. Hardening Your Deployment
6. Key Takeaways

Chapter 13: Deployment, APIs, and Real-World Integration

1. Chapter Dependencies and Project Structure
2. Packaging Agents as Services
3. RESTful Agent APIs
4. Web Interfaces and Dashboards
5. Desktop and System Integration
6. Production Deployment Patterns
7. Key Takeaways

Chapter 14: Capstone Project — Building a Production Coding Agent

1. Project Overview: The CodePilot Agent
2. Project Structure
3. Step 1: Configuration and Dependencies
4. Step 2: Tool Registry and Implementation
5. Step 3: Memory System Integration
6. Step 4: RAG Pipeline for Codebase Indexing
7. Step 5: Agent Core with Prompts and Reasoning
8. Step 6: Security Layer
9. Step 7: Observability Setup
10. Step 8: FastAPI Service Layer
11. Step 9: Docker Deployment
12. Step 10: Web Interface
13. Step 11: Testing
14. Running the Complete System
15. Architecture Summary
16. Key Takeaways

Conclusion: The Future of Local Agents

1. What We Have Built
2. The Trajectory Ahead
3. Principles That Will Endure
4. Key Takeaways

References

Appendix A: Ollama Environment Variables and Configuration Reference

1. Network and Binding Variables
2. Model Storage Variables
3. Concurrency and Queue Variables
4. GPU and Hardware Variables
5. Context and Inference Variables
6. Setting Environment Variables by Platform
7. Recommended Production Configurations

Appendix B: Troubleshooting Common GPU, Driver, and OOM Errors

1. Enabling Debug Logging
2. NVIDIA GPU Not Detected
3. AMD GPU Not Detected
4. CUDA Out of Memory (OOM) Errors
5. CPU Fallback (Silent or Explicit)
6. Model Loading Fails or Hangs
7. Connection Refused Errors
8. Apple Silicon Specific Issues
9. Diagnostic Checklist

Appendix C: Model Quantization Cheat Sheet

1. Quick Decision Guide
2. Quantization Level Comparison
3. Key Principles
4. VRAM Estimation Formula
5. Model Tag Naming Convention
6. Speed vs Quantization

Appendix D: Reusable Code Patterns

1. Pattern 1: Retry with Exponential Backoff and Jitter
2. Pattern 2: Circuit Breaker for External Services
3. Pattern 3: Structured JSON Logger
4. Pattern 4: Token Counter Utility
5. Pattern 5: Rate Limiter for API Calls
6. Pattern 6: Context Window Manager