Inside llama.cpp

The Complete Guide to Building, Running, and Optimizing Local LLM Inference

Introduction: Why Local Inference Matters

Chapter 1: The llama.cpp Revolution

1. The Birth of GGML
2. Whisper.cpp and the Catalyst Moment
3. The GGUF Format
4. The Open-Source LLM Landscape
5. Why Local Inference Matters

Chapter 2: Architecture Deep Dive

1. The GGML Tensor System and Computation Graph
2. The GGUF File Format: Structure and Semantics
3. Model Loading Pipeline: From Disk to VRAM
4. The Inference Engine: Token Generation Loop

Chapter 3: Building from Source — Linux

1. Prerequisites and Dependency Management
2. CMake Configuration Options for Linux
3. Building with CPU-Only Support
4. Enabling CUDA on Linux
5. Enabling ROCm on Linux
6. Common Build Errors and Fixes

Chapter 4: Building from Source — macOS and Windows

1. macOS: Apple Silicon Metal Builds (The Sweet Spot)
2. macOS: Intel Macs and CPU-Only Builds
3. Windows: MSVC Build with CMake
4. Windows: WSL2 as a Practical Alternative
5. Cross-Compilation Considerations
6. The XCFramework: Native iOS, visionOS, and tvOS Support

Chapter 5: Model Conversion and Quantization

1. The Conversion Pipeline: Hugging Face to GGUF
2. Quantization Theory: Why Quantize and What Is Lost
3. GGML Quantization Schemes Explained
4. Choosing the Right Quantization for Your Use Case
5. GPTQ and AWQ Model Support

Chapter 6: Inference Fundamentals — CLI Tools

1. llama-cli: The Interactive Inference Tool
2. Key Generation Parameters Explained
3. Prompt Formats and Chat Templates
4. Context Window Management and KV Cache Tuning
5. Streaming Output and Real-Time Token Generation

Chapter 7: Server Mode and API Integration

1. Starting and Configuring llama-server
2. The OpenAI-Compatible API Specification
3. Authentication, CORS, and Security Considerations
4. Integrating with Existing Applications
5. Router Mode and Dynamic Model Management

Chapter 8: Advanced Inference Techniques

1. Batch Inference: Multiple Prompts, Throughput Gains
2. Speculative Decoding: How It Works, When It Helps
3. Grammar-Constrained Generation: JSON, Regex, Structured Output
4. Advanced Sampling: Mirostat, Top-A, Tail-Free
5. KV Cache Optimizations: Offloading and Sliding Windows

Chapter 9: Embeddings and Multimodal Models

1. Embedding Generation with llama.cpp
2. Embedding Use Cases: RAG, Semantic Search, Clustering
3. Multimodal Model Support: LLaVA, Qwen2-VL, and Beyond
4. Image Preprocessing and Tokenization for Vision Models
5. Practical Multimodal Workflows

Chapter 10: GPU Backends and Hardware Acceleration

1. CUDA Backend: NVIDIA GPUs, Memory Management, Performance Tuning
2. ROCm Backend: AMD GPUs, Setup and Limitations
3. Vulkan Backend: Cross-Platform GPU Acceleration
4. Apple Metal: The macOS/iOS Sweet Spot
5. Multi-GPU Inference and Distributed Computing

Chapter 11: Performance Tuning and Benchmarking

1. llama-bench: The Benchmarking Tool
2. Measuring Tokens Per Second Across Hardware Configurations
3. Memory Footprint Analysis and Optimization
4. Throughput vs Latency Trade-offs in Production
5. Profiling Techniques and Identifying Bottlenecks

Chapter 12: Real-World Deployment and Production

1. Docker Containerization of llama.cpp
2. Docker Compose for Production Deployment
3. Kubernetes Deployment Patterns
4. Monitoring, Logging, and Alerting
5. CI/CD Pipelines for Model Serving
6. Case Studies: Production Deployments at Various Scales
7. Troubleshooting Common Production Issues

Conclusion: The Future of Local Inference

References