vLLM and the Engineering of Fast LLM Inference

A Systems Guide to High-Performance Large Language Model Serving

Introduction: The LLM Inference Problem

1. What Makes LLM Inference Hard
2. A Day in the Life of an Inference Request
3. Why vLLM Is Our Lens
4. How to Use This Book

Chapter 1: Transformer Inference from First Principles

1. The Transformer Forward Pass
2. Self-Attention Mechanics and Complexity
3. Prefill Versus Decode: Two Different Problems
4. Why Attention Dominates Decode Performance
5. The Role of Positional Embeddings
6. FlashAttention and Modern Attention Kernels

Chapter 2: GPU Architecture for LLM Inference

1. GPU Compute Units: SMs, CUDA Cores, and Tensor Cores
2. The GPU Memory Hierarchy: DRAM, SRAM, Registers
3. Memory Bandwidth as the Primary Bottleneck
4. Arithmetic Intensity and Roofline Analysis
5. PCIe, NVLink, and Interconnect Topology
6. CUDA Streams and Concurrency Primitives

Chapter 3: Measuring What Matters: Inference Metrics and Workloads

1. Latency Metrics: TTFT, ITL, and End-to-End Delay
2. Throughput Metrics: Tokens per Second and Requests per Second
3. GPU Utilization and Efficiency
4. Workload Profiles: Interactive, Batch, and Hybrid
5. Designing Reproducible Benchmarks
6. Cost Models: Tokens, GPUs, and Dollars

Chapter 4: The KV-Cache and Memory Management

1. Why the KV-Cache Exists
2. KV-Cache Memory: How Much and Why It Matters
3. Fragmentation and the Allocation Problem
4. PagedAttention: The Core Innovation
5. Block Tables and Virtual Memory for KV-Cache
6. Eviction Policies and Cache Thrashing

Chapter 5: Batching and Scheduling Fundamentals

1. Static Batching and Its Limitations
2. Continuous Batching: Keeping the GPU Busy
3. Request Scheduling Policies
4. Preemption and Speculative Scheduling
5. The Impact of Sequence Length Distribution
6. Admission Control and Concurrency Limits

Chapter 6: Inside vLLM: Architecture and Execution Model

1. The vLLM Process Model and Workers
2. Engine, Scheduler, and Executor Roles
3. The Request Lifecycle in vLLM
4. Model Runner and GPU Worker Implementation
5. Event Loop and CPU-GPU Coordination
6. How vLLM Handles the API Server

Chapter 7: vLLM’s PagedAttention Implementation

1. Virtual Block Allocation
2. Block Table Construction and Maintenance
3. The Attention Kernel with Paged KV-Cache
4. Physical Memory Management
5. Handling Variable Sequence Lengths
6. Interaction with the Scheduler

Chapter 8: Continuous Batching in vLLM

1. The Scheduler Loop
2. Running, Waiting, and Swapped States
3. Preemption Strategies
4. Chunked Prefill
5. Integration with PagedAttention
6. Tuning Scheduler Parameters

Chapter 9: vLLM Configuration and Deployment Basics

1. Installation: Dependencies, CUDA, and Compatibility
2. Launching Your First vLLM Server
3. OpenAI-Compatible API Configuration
4. Docker and Container Deployments
5. Model Selection and Loading Options
6. Basic Configuration Parameters

Chapter 10: Quantization for Inference

1. Why Quantize for Inference
2. Weight-Only Quantization: AWQ, GPTQ, INT8, FP8
3. Activation Quantization and Perplexity Impact
4. vLLM’s Quantization Backends
5. Choosing the Right Precision for Your Workload
6. Quantized Model Serving Examples

Chapter 11: Tensor Parallelism and Multi-GPU Scaling

1. The Limits of Single-GPU Inference
2. How Tensor Parallelism Works
3. vLLM’s Tensor Parallel Implementation
4. Communication Overhead and NVLink
5. Performance Scaling Curves
6. Configuration for Multi-GPU Deployments

Chapter 12: Pipeline Parallelism and Expert Parallelism

1. Pipeline Parallelism Basics
2. Pipeline Bubble and Scheduling Challenges
3. vLLM’s Pipeline Parallelism Support
4. Mixture of Experts and Expert Parallelism
5. DeepSeek and Qwen MoE Models in vLLM
6. Hybrid Parallelism Strategies

Chapter 13: Distributed Inference Across Nodes

1. Multi-Node Communication with Ray
2. Network Topology and Bandwidth Planning
3. Fault Tolerance Across Nodes
4. Load Balancing Strategies
5. Kubernetes Deployments with vLLM
6. Operational Complexity of Distributed Serving

Chapter 14: Caching, Prefix Caching, and Context Optimization

1. Prefix Caching in vLLM
2. Hash-Based Cache Lookup
3. Cache Validity and Model Versioning
4. System Prompts and Repeated Contexts
5. Multi-Query and Grouped-Query Attention
6. Long-Context Optimization Techniques

Chapter 15: Speculative Decoding

1. The Speculative Decoding Algorithm
2. Draft Model Selection
3. Token Tree Speculation
4. vLLM’s Speculative Decoding Implementation
5. Measuring Acceptance Rates and Speedup
6. When Speculative Decoding Helps and Hurts

Chapter 16: CUDA Graphs and Kernel Optimization

1. CUDA Kernel Launch Overhead
2. CUDA Graphs in vLLM
3. Attention Backend Selection
4. FlashAttention Integration
5. Profiling Kernel Performance
6. When Low-Level Optimization Matters

Chapter 17: Production Serving Engineering

1. Health Checks and Readiness Probes
2. Autoscaling Strategies
3. Rate Limiting and Admission Control
4. Observability: Metrics, Logs, and Tracing
5. Authentication and API Security
6. Rolling Upgrades and Model Versioning

Chapter 18: Performance Engineering and Tuning

1. Establishing Performance Baselines
2. Profiling CPU and GPU with vLLM
3. Diagnosing Bottlenecks
4. Tuning for Interactive Workloads
5. Tuning for Batch Workloads
6. Configuration Decision Trees

Chapter 19: Troubleshooting and Failure Modes

1. Out-of-Memory Failures
2. CUDA and Driver Incompatibilities
3. Scheduling and Throughput Issues
4. GPU Utilization Problems
5. Distributed System Failures
6. Version and API Compatibility Issues

Chapter 20: The Ecosystem: vLLM Versus Alternatives

1. Hugging Face Text Generation Inference
2. TensorRT-LLM: NVIDIA’s Approach
3. SGLang and Structured Generation
4. llama.cpp and CPU/Edge Inference
5. Feature Matrix and Workload Mapping

Chapter 21: The Economics and Future of LLM Inference

1. Hardware Economics: GPUs and Inference Cost
2. Capacity Planning Methodology
3. Emerging Techniques and Trends
4. What Will Remain Stable
5. Architectural Recommendations

Conclusion: Engineering Judgment in Inference

1. The Inference Engineering Mindset
2. Key Principles That Endure
3. Building Versus Buying
4. A Final Architecture Recommendation

Back Matter

1. Glossary of Key Terms
2. Quick Reference: Key vLLM Parameters
3. Performance Tuning Checklist
4. References