Building Low-Latency LLM Infrastructure

From Fundamentals to Production-Grade Systems

Introduction

1. Who This Book Is For
2. How This Book Is Organized
3. A Note on Tools and Versioning
4. How to Read This Book

Chapter 1: The Anatomy of a Single Request

1. From HTTP Request to Generated Token: The Complete Lifecycle
2. Tokenization: The Hidden Cost Before Computation Begins
3. Prefill: Processing the Input Prompt in One Pass
4. Decoding: Generating Tokens One at a Time
5. The Key-Value Cache: Trading Memory for Recomputation
6. Latency Budgets: TTFT, TPOT, and What They Mean for Users
7. Summary

Chapter 2: Measuring What Matters

1. Latency Metrics That Actually Inform Decisions
2. Throughput vs Latency: The Fundamental Trade-off
3. Utilization Metrics: GPU Compute and Memory Bandwidth
4. Queueing Theory for Inference Systems
5. Profiling Fundamentals: Traces, Metrics, and Flame Graphs
6. Building a Baseline Benchmark Harness
7. Summary

Chapter 3: GPU Architecture for Inference

1. The GPU as a Parallel Compute Engine
2. Memory Hierarchy: Registers, Shared Memory, Global Memory, HBM
3. CUDA Concepts Relevant to Inference
4. Why LLMs Are Memory-Bound (Mostly)
5. The NVIDIA GPU Lineup for Inference
6. Beyond NVIDIA: AMD, Intel, and Custom Silicon
7. Summary

Chapter 4: The CPU-GPU Interface

1. Data Movement Patterns in Inference
2. PCIe Bandwidth and Latency
3. Pinned Memory and Asynchronous Transfers
4. Zero-Copy and Unified Memory Considerations
5. Kernel Launch Overhead and CUDA Graphs
6. Summary

Chapter 5: Batching and Scheduling

1. Static Batching: Simple but Wasteful
2. Continuous Batching (vLLM-style): The Breakthrough
3. Dynamic Batching and Micro-batching
4. Scheduling Policies: Fairness, Priority, and Latency
5. Preemption and Speculative Execution
6. Implementing a Scheduler: Design Decisions
7. Summary

Chapter 6: Memory Management for LLMs

1. KV Cache Size Calculations and Capacity Planning
2. PagedAttention and Block Management (vLLM)
3. Prefix Caching for Shared Contexts
4. KV Cache Eviction Policies
5. Quantization: Compressing Weights and Activations
6. Memory Fragmentation and Defragmentation
7. Summary

Chapter 7: Advanced Decoding Strategies

1. Sampling Methods and Their Latency Impact
2. Speculative Decoding: Drafting Faster Than Verifying
3. Medusa and Multi-Token Prediction Heads
4. Early Exit and Layer Skipping
5. Repetition Handling and Stopping Criteria
6. Summary

Chapter 8: Parallelism Strategies for Large Models

1. Tensor Parallelism: Splitting Matrices Across GPUs
2. Pipeline Parallelism: Splitting Layers Across GPUs
3. Data Parallelism for Inference: When and Why
4. Sequence (Context) Parallelism: Splitting Long Prompts
5. Expert Parallelism for Mixture-of-Experts Models
6. Hybrid Parallelism: Combining Strategies
7. Summary

Chapter 9: Multi-GPU and Multi-Node Systems

1. NVLink and NVSwitch: Intra-Node Communication
2. InfiniBand and RoCE: Inter-Node Communication
3. NCCL and Collective Communication
4. Topology-Aware Placement and Scheduling
5. RPC Frameworks for Distributed Serving
6. Fault Tolerance in Distributed Inference
7. Summary

Chapter 10: Building an Inference Server

1. From Single-Model Script to Production Server
2. API Design: OpenAI-Compatible Endpoints and Beyond
3. Streaming Responses: Why and How
4. Choosing an Inference Engine: vLLM vs SGLang vs TensorRT-LLM vs llama.cpp
5. Containerization and Deployment Basics
6. Warmup, Preflight, and Readiness Probes
7. Summary

Chapter 11: Cluster-Scale Serving

1. Load Balancing Strategies for Inference Traffic
2. Request Routing: Model Selection and Workload Segmentation
3. Admission Control and Backpressure
4. Autoscaling: Signals, Policies, and Pitfalls
5. Multi-Model Serving on Shared Infrastructure
6. Kubernetes and GPU Orchestration
7. Summary

Chapter 12: Disaggregated Architectures

1. Why Disaggregate Prefill and Decode?
2. Architecture of a Disaggregated System
3. Implementations and Research
4. Trade-offs and Complexity
5. Distributed Caching Layers
6. Summary

Chapter 13: Reliability and Observability

1. Health Checking and Readiness for GPU Services
2. Distributed Tracing for Inference Requests
3. Metrics That Matter in Production
4. Graceful Degradation Under Load
5. Retry Logic, Timeouts, and Circuit Breakers
6. Incident Response and Debugging GPU Failures
7. Summary

Chapter 14: Capacity Planning and Cost Optimization

1. Workload Characterization: Understanding Your Traffic
2. Capacity Calculations: From Tokens to GPUs
3. Cost Per Token and Cost Per Request Modeling
4. Right-Sizing Hardware for Your Workload
5. Performance Regression Testing
6. Summary

Chapter 15: Extremely Low-Latency Serving

1. Sub-100ms TTFT: Is It Achievable?
2. Real-Time Conversational and Voice Workloads
3. CUDA Graphs and Kernel Fusion at Scale
4. Communication/Computation Overlap in Distributed Systems
5. Edge and On-Device Inference for Ultra-Low Latency
6. Summary

Chapter 16: Specialized Workloads and Emerging Techniques

1. Long-Context Inference (100K+ Tokens)
2. Reasoning Models with Variable Output Lengths
3. Multimodal Inference: Images, Audio, and Video
4. High-Concurrency Serving Patterns
5. Heterogeneous GPU Fleets
6. Inference-Aware Model Optimization
7. Summary

Chapter 17: End-to-End Reference Architectures

1. Architecture A: Single GPU Prototype (Proof of Concept)
2. Architecture B: Single Multi-GPU Server (Small Team Production)
3. Architecture C: Small Cluster (Startup/SMB Production)
4. Architecture D: Geographically Distributed Platform (Enterprise Scale)
5. Summary

Chapter 18: Designing Low-Latency LLM Systems from First Principles

1. Requirements Gathering and Workload Characterization
2. Latency Budget Construction
3. Architecture Selection Decision Tree
4. Hardware Sizing and Procurement Strategy
5. Implementation Sequencing: Build in Stages
6. Optimization Priorities: Where to Spend Your Time
7. Production Readiness Checklist
8. Summary

Chapter 19: Conclusion: Enduring Principles for a Fast-Changing Field

1. The Core Mental Models
2. Where LLM Infrastructure Is Heading
3. A Note on Engineering Philosophy
4. Closing Thoughts

References