Practical LLM Inference

Quantization, GGUF and Local Models

Introduction: The Local Inference Stack

1. Why Engineers Care About Local Inference
2. What This Book Will Not Teach You
3. The Real Bottleneck: Memory Bandwidth, Not Compute
4. How to Read This Book

Chapter 1: Transformer Inference Fundamentals

1. From Training to Inference: What Changes
2. The Forward Pass Anatomy
3. Self-Attention and Its Costs
4. Why Autoregression Matters
5. Arithmetic Intensity and Compute Density

Chapter 2: Tokenization, Prefill, and Decode

1. How Tokenizers Actually Work
2. The Prefill Phase: Parallel Computation
3. The Decode Phase: Sequential Bottleneck
4. Context Length and Its Real Cost
5. Why These Phases Demand Different Optimizations

Chapter 3: Model Weights, Formats, and GGUF Architecture

1. From Checkpoints to Inference: Weight Formats
2. The GGML Story and Design Goals
3. GGUF Header Structure and Metadata
4. Tensor Storage Layout and Quantization Types
5. Reading a GGUF File: A Practical Walkthrough

Chapter 4: Quantization: Mathematics, Methods, and Tradeoffs

1. Why Quantization Works on Transformers
2. Symmetric vs Asymmetric Quantization
3. Per-Tensor, Per-Channel, Per-Group Schemes
4. GGUF Quantization Types Decoded
5. Measuring Quality Loss: Benchmarks and Perception
6. Choosing a Quant Level for Your Use Case

Chapter 5: Memory, Bandwidth, and Hardware Constraints

1. Computing Model Size: The Exact Formula
2. KV-Cache Memory Requirements
3. Memory Bandwidth vs Compute Throughput
4. CPU Architectures for Inference
5. GPU Architectures and VRAM Realities
6. Hardware Selection Decision Framework

Chapter 6: llama.cpp: Installation, Models, and Core Usage

1. Building llama.cpp from Source
2. Installing Prebuilt Binaries
3. Acquiring Models: Hugging Face and Beyond
4. Converting Weights to GGUF
5. Quantizing Models with llama.cpp
6. Your First Inference Run

Chapter 7: GPU Acceleration and Offloading Strategies

1. How Offloading Works Under the Hood
2. Full vs Partial Offloading: When Each Wins
3. CUDA Backend Configuration
4. Vulkan and Cross-Vendor GPU Support
5. Apple Silicon and Metal Acceleration
6. Diagnosing Offloading Failures

Chapter 8: KV-Cache, Context Windows, and Memory Management

1. How the KV-Cache Works
2. Sizing Your KV-Cache Correctly
3. Context Overflow and Handling Strategies
4. RoPE Scaling and Long-Context Techniques
5. Cache Eviction and Sliding Windows
6. Memory Fragmentation and Allocation Patterns

Chapter 9: Throughput, Batching, and Concurrency

1. Single Request vs Concurrent Serving
2. Static Batching Mechanics
3. Continuous Batching and Scheduling Strategies
4. Queue Management Under Load
5. Configuring llama.cpp Server for Throughput
6. Capacity Planning and Dimensioning

Chapter 10: Benchmarking, Profiling, and Bottleneck Diagnosis

1. Metrics That Matter: Definitions and Formulas
2. Designing Valid Benchmark Experiments
3. Profiling CPU Utilization and Vector Units
4. GPU Profiling and Occupancy Analysis
5. Memory Bandwidth Measurement Techniques
6. Systematic Bottleneck Diagnosis Workflow

Chapter 11: Production Systems: APIs, RAG, and Deployment

1. Building a Private Local Assistant
2. Coding Assistant Integration Patterns
3. RAG Service Architecture
4. Multi-User Inference API Design
5. Containerization and Orchestration
6. Observability, Security, and Reliability

Chapter 12: Troubleshooting, Comparisons, and the Road Ahead

1. Common Failure Modes and Diagnostics
2. Performance Troubleshooting Decision Tree
3. llama.cpp vs Other Inference Runtimes
4. What Is Durable vs Rapidly Changing
5. Emerging Trends and Open Problems

Conclusion: Engineering Judgment Over Magic Numbers

References