Inside llama.cpp

The Complete Guide to Building, Running and Optimizing Local LLM Inference

Introduction: Why Local Inference Matters

Chapter 1: The llama.cpp Revolution

1. The Birth of GGML
2. Whisper.cpp and the Catalyst Moment
3. The GGUF Format
4. The Open-Source LLM Landscape
5. Why Local Inference Matters

Chapter 2: Architecture Deep Dive

1. The GGML Tensor System and Computation Graph
2. The GGUF File Format: Structure and Semantics
3. Model Loading Pipeline: From Disk to VRAM
4. The Inference Engine: Token Generation Loop

Chapter 3: Inside the GGML Tensor Library

1. The Define-Then-Execute Computation Model
2. The ggml_tensor Structure
3. Memory Arenas and Contexts
4. Backend Abstraction and Plugin System
5. CPU Backend: SIMD Dispatch and Quantized Kernels
6. Flash Attention Implementation in GGML
7. From GGML to llama.cpp: The Integration Layer

Chapter 4: Custom GGML Backend Development and Novel Hardware

1. When to Write a Custom Backend
2. The GGML Backend Interface
3. Step-by-Step: Implementing a Minimal Backend
4. Writing Efficient Custom Kernels
5. Debugging and Testing Your Backend
6. Contributing to llama.cpp

Chapter 5: The llama.cpp C/C++ API

1. Overview of llama.h
2. Loading a Model
3. Initializing a Context
4. Tokenization and Building Batches
5. Single-Token Generation and Sampling
6. Common Sampler Operations
7. Saving and Loading Sessions
8. Building Applications with the C API

Chapter 6: Building from Source - Linux

1. Prerequisites and Dependency Management
2. CMake Configuration Options for Linux
3. Building with CPU-Only Support
4. Enabling CUDA on Linux
5. Enabling ROCm on Linux
6. Common Build Errors and Fixes

Chapter 7: Building from Source - macOS and Windows

1. macOS: Apple Silicon Metal Builds (The Sweet Spot)
2. macOS: Intel Macs and CPU-Only Builds
3. Windows: MSVC Build with CMake
4. Windows: WSL2 as a Practical Alternative
5. Cross-Compilation Considerations
6. The XCFramework: Native iOS, visionOS and tvOS Support

Chapter 8: Model Conversion and Quantization

1. The Conversion Pipeline: Hugging Face to GGUF
2. Quantization Theory: Why Quantize and What Is Lost
3. GGML Quantization Schemes Explained
4. Choosing the Right Quantization for Your Use Case
5. GPTQ and AWQ Model Support

Chapter 9: LoRA Adapters Deep Dive

1. What Is LoRA and Why It Matters
2. Converting PEFT Adapters to GGUF
3. Loading Adapters with the CLI
4. Using LoRA in llama-server
5. Using LoRA with the C API
6. Stacking Multiple Adapters
7. Memory and Performance Considerations
8. LoRA vs Full Fine-Tuning

Chapter 10: Inference Fundamentals - CLI Tools

1. llama-cli: The Interactive Inference Tool
2. Key Generation Parameters Explained
3. Prompt Formats and Chat Templates
4. Context Window Management and KV Cache Tuning
5. Streaming Output and Real-Time Token Generation

Chapter 11: Server Mode and API Integration

1. Starting and Configuring llama-server
2. The OpenAI-Compatible API Specification
3. Authentication, CORS and Security Considerations
4. Tool Calling and MCP Server Integration
5. Integrating with Existing Applications
6. Router Mode and Dynamic Model Management

Chapter 12: Server Internals and Scheduling

1. How Continuous Batching Works
2. The Batch Processing Pipeline
3. Decode-Maximal Scheduling
4. Request Lifecycle in the Server
5. KV Cache and Context Management
6. Router Mode Internals
7. Performance Characteristics

Chapter 13: Security, Privacy and Air-Gapped Deployments

1. The Local Inference Security Model
2. Authentication and Authorization
3. Network Security and SSL/TLS
4. Container Security
5. Rate Limiting and DoS Protection
6. Air-Gapped Deployment
7. Model Supply Chain Security
8. Data Privacy and Prompt Redaction
9. Defense in Depth Checklist

Chapter 14: Advanced Inference Techniques

1. Batch Inference: Multiple Prompts, Throughput Gains
2. Speculative Decoding: How It Works, When It Helps
3. Grammar-Constrained Generation: JSON, Regex, Structured Output
4. Advanced Sampling: Mirostat, Top-A, Tail-Free
5. KV Cache Optimizations: Offloading and Sliding Windows

Chapter 15: Embeddings and Multimodal Models

1. Embedding Generation with llama.cpp
2. Embedding Use Cases: RAG, Semantic Search, Clustering
3. Multimodal Model Support: LLaVA, Qwen2-VL and Beyond
4. Image Preprocessing and Tokenization for Vision Models
5. Practical Multimodal Workflows

Chapter 16: GPU Backends and Hardware Acceleration

1. CUDA Backend: NVIDIA GPUs, Memory Management, Performance Tuning
2. ROCm Backend: AMD GPUs, Setup and Limitations
3. Vulkan Backend: Cross-Platform GPU Acceleration
4. Apple Metal: The macOS/iOS Sweet Spot
5. Multi-GPU Inference and Distributed Computing

Chapter 17: Performance Tuning and Benchmarking

1. llama-bench: The Benchmarking Tool
2. Measuring Tokens Per Second Across Hardware Configurations
3. Memory Footprint Analysis and Optimization
4. Throughput vs Latency Trade-offs in Production
5. Profiling Techniques and Identifying Bottlenecks

Chapter 18: Ecosystem Tools and Integrations

1. Ollama: The Popular Abstraction Layer
2. LM Studio: Desktop GUI with llama.cpp Backend
3. koboldcpp: Specialized Backend for Creative Writing
4. llama-cpp-python: Python Bindings for Direct Integration
5. llama-cpp-rs: Rust Bindings
6. WebGPU and Browser-Based Inference
7. Choosing the Right Tool

Chapter 19: Real-World Deployment and Production

1. Docker Containerization of llama.cpp
2. Docker Compose for Production Deployment
3. Kubernetes Deployment Patterns
4. Monitoring, Logging and Alerting
5. CI/CD Pipelines for Model Serving
6. Case Studies: Production Deployments at Various Scales
7. Troubleshooting Common Production Issues

Conclusion: The Future of Local Inference

References