Local-First AI Engineering

Running Agents Without the Cloud

Introduction: The Case for Local-First AI

1. The Cloud Dependency Problem
2. Defining Local-First AI
3. What This Book Is and Is Not
4. The Reference Project

Chapter 1: Foundations - Hardware, Linux and the Local AI Stack

1. The Local AI Hardware Stack
2. GPUs and Accelerators
3. Hardware Discovery and Monitoring on Linux
4. Linux as the AI Operating Foundation
5. GPU Drivers and Accelerator Ecosystems
6. Practical Hardware Sizing Methodology
7. Chapter Summary

Chapter 2: Transformer Inference from First Principles

1. Tokenization and Embeddings
2. Attention Mechanics and KV Caches
3. Prefill Versus Decode: Two Different Computational Regimes
4. Batching Strategies
5. Sampling and Decoding Controls
6. Memory and Compute Accounting
7. Chapter Summary

Chapter 3: Model Selection and Formats

1. Architecture Families
2. Model Roles
3. Model Formats and File Layouts
4. Repositories and Provenance
5. Licensing and Compliance
6. Selection Methodology
7. Chapter Summary

Chapter 4: Quantization and Inference Optimization

1. Numerical Representations
2. Quantization Approaches
3. GGUF Quantization Families
4. CPU/GPU Offloading Strategies
5. Advanced Optimizations
6. Systematic Benchmarking of Optimizations
7. Chapter Summary

Chapter 5: Local Inference Runtimes and Serving

1. llama.cpp and the GGUF Ecosystem
2. Ollama as a Managed Local Runtime
3. vLLM for High-Throughput Local Serving
4. Hugging Face Transformers and Related Libraries
5. ONNX Runtime and Alternative Backends
6. Building the Reference Model Server
7. Chapter Summary

Chapter 6: APIs, Interfaces and Interoperability

1. API Design for Local Inference
2. OpenAI-Compatible API as Interoperability Convention
3. Streaming Responses and Backpressure
4. Structured Outputs and JSON Schemas
5. Tool/Function Calling Interfaces
6. Authentication, Rate Limiting and Concurrency Control
7. Health Checks, Graceful Shutdown and Service Discovery
8. Chapter Summary

Chapter 7: Local Agent Architecture

1. What Is an Agent
2. The Agent Loop
3. Structured Tool Calling
4. State Management and Memory
5. Planning and Task Decomposition
6. Multi-Agent Architectures
7. Chapter Summary

Chapter 8: Building Fully Local Agents

1. Minimal Local Model Client
2. Tool-Using Assistant
3. Local Document Analysis System
4. Local Coding Assistant
5. Research/Knowledge Agents Over Local Corpora
6. Background Autonomous Services
7. Chapter Summary

Chapter 9: Local RAG, Memory and Knowledge Systems

1. Document Ingestion Pipelines
2. Embedding Models for Local Use
3. Vector Search and Indexes
4. Hybrid Retrieval
5. Reranking
6. Context Assembly and Citations
7. Persistent Storage: SQLite and Structured Stores
8. Conversation Memory and Episodic Storage
9. Indexing Pipelines and Incremental Updates
10. Retrieval Evaluation
11. Defenses Against Poisoned Content
12. Chapter Summary

Chapter 10: Security and Trust Boundaries for Local AI

1. The Local Fallacy
2. Model Provenance and Artifact Verification
3. Software Supply Chain Security
4. Prompt Injection Defenses
5. Tool Abuse Prevention
6. Linux Security Controls
7. Network Security
8. Authentication and Access Control
9. Secrets Management
10. Audit Logging
11. Human Approval Gates
12. Kill Switches
13. Chapter Summary

Chapter 11: Offline and Air-Gapped Operation

1. Dependency Acquisition Strategy
2. Reproducible Package Mirrors
3. Model Transfer and Integrity Verification
4. Offline Documentation
5. Software Updates and Patch Management
6. License Tracking and Compliance
7. Backup and Recovery
8. Time Synchronization
9. Secure Data Import/Export
10. Maintaining Long-Term Disconnected Deployments
11. Chapter Summary

Chapter 12: Performance Engineering and Benchmarking

1. Why Benchmarking Matters for Local AI
2. Benchmark Metrics Defined
3. Building Reproducible Benchmark Harnesses
4. Interpreting Benchmark Results
5. Establishing Service-Level Objectives
6. Capacity Planning
7. Chapter Summary

Chapter 13: Reliability Engineering for Autonomous Local Systems

1. Process Supervision
2. Health Checks
3. Retries with Bounded Policies
4. Timeouts
5. Circuit Breakers
6. Queue Management and Backpressure
7. Crash Recovery and Checkpointing
8. GPU Out-of-Memory Handling
9. Degraded-Mode Operation
10. Chapter Summary

Chapter 14: Observability and Evaluation

1. Structured Logging
2. Metrics Collection
3. Distributed Tracing
4. GPU and Hardware Monitoring
5. Local Dashboards
6. Alerting
7. Behavioral Evaluation
8. Testing Strategies
9. Continuous Evaluation
10. Chapter Summary

Chapter 15: Deployment Patterns and Networking

1. Developer Laptop Deployment
2. Gaming-Class Workstation Deployment
3. Dedicated Linux Inference Server
4. Multi-GPU Workstation/Server
5. Home Lab Deployment
6. Small Office Deployment
7. Edge Appliance Deployment
8. Air-Gapped Environment Deployment
9. Centralized Versus Distributed Inference
10. Networking for Local AI Systems
11. Chapter Summary

Chapter 16: Integration, Lifecycle Operations and Architecture Decisions

1. Integration with Existing Software
2. Lifecycle and Operations
3. Local Versus Cloud Versus Hybrid: Decision Frameworks
4. Chapter Summary

Conclusion: The LocalForge Platform in Practice

1. The Complete Architecture
2. Getting Started: From Zero to Working System
3. Adapting to Your Environment
4. Future Directions in Local AI
5. Final Thoughts

References