Leanpub Header

Skip to main content

Local-First AI Engineering

Running Agents Without the Cloud

Local-First AI Engineering
This book is 100% completeLast updated on 2026-08-26

Build AI that keeps working when the cloud doesn't. Local-First AI Engineering shows you how to run capable, private AI systems on infrastructure you control. From hardware and inference to RAG, agents, security and production ops, you'll learn how to build local AI that is fast, reliable and truly yours.

Minimum price

$25.00

$35.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
359
Pages
About

About

About the Book

A comprehensive systems-engineering handbook for building AI applications and autonomous agents that remain capable, private, controllable, secure, maintainable, reproducible, and operational even when the cloud is unavailable or intentionally absent. This book takes you from hardware selection through Linux configuration, inference optimization, RAG pipelines, agent architecture, security hardening, reliability engineering, and production operations — with a complete reference implementation that evolves across every chapter. Designed for software engineers, AI/ML engineers, systems engineers, platform engineers, DevOps/SRE practitioners, security engineers, researchers, technical architects, and advanced developers who need to build real local-first AI systems they own end-to-end.

Author

About the Author

Steve Publications

Steve is a technology professional with more than 20 years of experience in software development, server infrastructure, cybersecurity, vulnerability research and reverse engineering. Throughout his career, he has designed, secured, analyzed and tested complex software and infrastructure, with a particular focus on understanding how systems fail and how they can be made more secure.

Outside of work, Steve enjoys sharing knowledge with the technology community. He collaborates with researchers, industry experts and technology professionals to write practical books covering software development, cybersecurity, cloud computing, networking, DevOps, artificial intelligence and enterprise technologies. His books focus on practical learning through clear explanations, real-world examples and hands-on exercises. With more than two decades of industry experience, his goal is to help IT professionals, students and technology enthusiasts build useful skills and stay current in a rapidly changing industry.

We believe readers deserve to know how our books are created. Most of our authors are not native English speakers, so we use AI to help translate, proofread manuscripts, fix grammar, improve sentence structure and make technical explanations easier to read. AI is used as an editing tool only. It does not replace the research, technical knowledge or hands-on experience behind our books. Some of our authors also prefer to remain anonymous for privacy or professional reasons. In those cases, we publish their work under a different name. The author's name may be different, but the quality of the content and our review process remain the same.

Every book is written, reviewed and maintained by experienced technology professionals, with contributions from our private technical community of more than 420 engineers and researchers. We spend far more time validating technical accuracy and keeping our content up to date than generating text. We are always interested in working with experienced professionals who have deep expertise in a particular technology or domain. If you would like to publish a book with us or help review an existing manuscript, we'd love to hear from you. Send us a message describing your area of expertise. We are especially interested in niche technologies, specialized skills and emerging topics that are underrepresented in existing technical literature.

If you look through the contents of our books, you'll see practical examples, detailed explanations and material that is regularly updated. Our goal is to publish books that professionals can actually rely on, not low-effort AI-generated content. If you ever feel that one of our books does not meet that standard, Leanpub offers a 60-day money-back guarantee. Feel free to request a refund if you are not satisfied with your purchase.

Contents

Table of Contents

Running Agents Without the Cloud

Introduction: The Case for Local-First AI

  1. The Cloud Dependency Problem
  2. Defining Local-First AI
  3. What This Book Is and Is Not
  4. The Reference Project

Chapter 1: Foundations — Hardware, Linux, and the Local AI Stack

  1. The Local AI Hardware Stack
  2. GPUs and Accelerators
  3. Hardware Discovery and Monitoring on Linux
  4. Linux as the AI Operating Foundation
  5. GPU Drivers and Accelerator Ecosystems
  6. Practical Hardware Sizing Methodology
  7. Chapter Summary

Chapter 2: Transformer Inference from First Principles

  1. Tokenization and Embeddings
  2. Attention Mechanics and KV Caches
  3. Prefill Versus Decode: Two Different Computational Regimes
  4. Batching Strategies
  5. Sampling and Decoding Controls
  6. Memory and Compute Accounting
  7. Chapter Summary

Chapter 3: Model Selection and Formats

  1. Architecture Families
  2. Model Roles
  3. Model Formats and File Layouts
  4. Repositories and Provenance
  5. Licensing and Compliance
  6. Selection Methodology
  7. Chapter Summary

Chapter 4: Quantization and Inference Optimization

  1. Numerical Representations
  2. Quantization Approaches
  3. GGUF Quantization Families
  4. CPU/GPU Offloading Strategies
  5. Advanced Optimizations
  6. Systematic Benchmarking of Optimizations
  7. Chapter Summary

Chapter 5: Local Inference Runtimes and Serving

  1. llama.cpp and the GGUF Ecosystem
  2. Ollama as a Managed Local Runtime
  3. vLLM for High-Throughput Local Serving
  4. Hugging Face Transformers and Related Libraries
  5. ONNX Runtime and Alternative Backends
  6. Building the Reference Model Server
  7. Chapter Summary

Chapter 6: APIs, Interfaces, and Interoperability

  1. API Design for Local Inference
  2. OpenAI-Compatible API as Interoperability Convention
  3. Streaming Responses and Backpressure
  4. Structured Outputs and JSON Schemas
  5. Tool/Function Calling Interfaces
  6. Authentication, Rate Limiting, and Concurrency Control
  7. Health Checks, Graceful Shutdown, and Service Discovery
  8. Chapter Summary

Chapter 7: Local Agent Architecture

  1. What Is an Agent
  2. The Agent Loop
  3. Structured Tool Calling
  4. State Management and Memory
  5. Planning and Task Decomposition
  6. Multi-Agent Architectures
  7. Chapter Summary

Chapter 8: Building Fully Local Agents

  1. Minimal Local Model Client
  2. Tool-Using Assistant
  3. Local Document Analysis System
  4. Local Coding Assistant
  5. Research/Knowledge Agents Over Local Corpora
  6. Background Autonomous Services
  7. Chapter Summary

Chapter 9: Local RAG, Memory, and Knowledge Systems

  1. Document Ingestion Pipelines
  2. Embedding Models for Local Use
  3. Vector Search and Indexes
  4. Hybrid Retrieval
  5. Reranking
  6. Context Assembly and Citations
  7. Persistent Storage: SQLite and Structured Stores
  8. Conversation Memory and Episodic Storage
  9. Indexing Pipelines and Incremental Updates
  10. Retrieval Evaluation
  11. Defenses Against Poisoned Content
  12. Chapter Summary

Chapter 10: Security and Trust Boundaries for Local AI

  1. The Local Fallacy
  2. Model Provenance and Artifact Verification
  3. Software Supply Chain Security
  4. Prompt Injection Defenses
  5. Tool Abuse Prevention
  6. Linux Security Controls
  7. Network Security
  8. Authentication and Access Control
  9. Secrets Management
  10. Audit Logging
  11. Human Approval Gates
  12. Kill Switches
  13. Chapter Summary

Chapter 11: Offline and Air-Gapped Operation

  1. Dependency Acquisition Strategy
  2. Reproducible Package Mirrors
  3. Model Transfer and Integrity Verification
  4. Offline Documentation
  5. Software Updates and Patch Management
  6. License Tracking and Compliance
  7. Backup and Recovery
  8. Time Synchronization
  9. Secure Data Import/Export
  10. Maintaining Long-Term Disconnected Deployments
  11. Chapter Summary

Chapter 12: Performance Engineering and Benchmarking

  1. Why Benchmarking Matters for Local AI
  2. Benchmark Metrics Defined
  3. Building Reproducible Benchmark Harnesses
  4. Interpreting Benchmark Results
  5. Establishing Service-Level Objectives
  6. Capacity Planning
  7. Chapter Summary

Chapter 13: Reliability Engineering for Autonomous Local Systems

  1. Process Supervision
  2. Health Checks
  3. Retries with Bounded Policies
  4. Timeouts
  5. Circuit Breakers
  6. Queue Management and Backpressure
  7. Crash Recovery and Checkpointing
  8. GPU Out-of-Memory Handling
  9. Degraded-Mode Operation
  10. Chapter Summary

Chapter 14: Observability and Evaluation

  1. Structured Logging
  2. Metrics Collection
  3. Distributed Tracing
  4. GPU and Hardware Monitoring
  5. Local Dashboards
  6. Alerting
  7. Behavioral Evaluation
  8. Testing Strategies
  9. Continuous Evaluation
  10. Chapter Summary

Chapter 15: Deployment Patterns and Networking

  1. Developer Laptop Deployment
  2. Gaming-Class Workstation Deployment
  3. Dedicated Linux Inference Server
  4. Multi-GPU Workstation/Server
  5. Home Lab Deployment
  6. Small Office Deployment
  7. Edge Appliance Deployment
  8. Air-Gapped Environment Deployment
  9. Centralized Versus Distributed Inference
  10. Networking for Local AI Systems
  11. Chapter Summary

Chapter 16: Integration, Lifecycle Operations, and Architecture Decisions

  1. Integration with Existing Software
  2. Lifecycle and Operations
  3. Local Versus Cloud Versus Hybrid: Decision Frameworks
  4. Chapter Summary

Conclusion: The LocalForge Platform in Practice

  1. The Complete Architecture
  2. Getting Started: From Zero to Working System
  3. Adapting to Your Environment
  4. Future Directions in Local AI
  5. Final Thoughts

References

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub