Kubernetes AI

Kubernetes AI

How to use this book

Prerequisites and pre-work

1. Want the recipes kept current?

Installing and Prerequisites

1. What you need
2. Choose a lab cluster
3. Install kubectl
4. Install Helm
5. Confirm permissions
6. Prepare the book namespace
7. Prepare a Hugging Face token
8. Check for GPU capacity
9. Add the NVIDIA Helm repository
10. Quick readiness test
11. What not to install yet
12. Try it yourself

Why Kubernetes for AI

1. Three shapes of AI workload
2. What makes AI workloads different
3. The cloud-native AI landscape
4. The running example: Mistral-7B in ai-lab
5. A map of the book
6. Going deeper
7. Try it yourself

GPU Foundations: The NVIDIA GPU Operator

1. What a node needs before nvidia.com/gpu: 1 works
2. Install the GPU Operator with Helm
3. ClusterPolicy: one CRD to rule the stack
4. GPU Feature Discovery: labels the scheduler can use
5. Verify with a CUDA vector-add pod
6. Driver upgrades without drama
7. Going deeper
8. Try it yourself

Scheduling & Sharing GPUs

1. How nvidia.com/gpu actually behaves
2. Steering pods to the right GPU
3. Sharing one GPU
4. DRA: where GPU allocation is going
5. Going deeper
6. Try it yourself

Serving Your First LLM with vLLM

1. Why vLLM
2. What fits where
3. The Deployment
4. Talk to your model
5. When the first run fails
6. Going deeper
7. Try it yourself

The Model-Serving Landscape

1. The contenders at a glance
2. Triton: the multi-model workhorse
3. TensorRT-LLM: paying for peak performance
4. NIM: the packaged option
5. KServe: the platform layer
6. Choosing without regret
7. Going deeper
8. Try it yourself

Optimizing Inference: Throughput, Latency, Cost

1. Continuous batching, properly understood
2. Quantization: the memory lever
3. Prefix caching: stop recomputing the same prompt
4. Speculative decoding: the latency lever
5. One base model, many fine-tunes: LoRA serving
6. Autoscaling done right
7. Going deeper
8. Try it yourself

Distributed Inference: Beyond One GPU

1. The math: when one GPU isn’t enough
2. Two ways to split a model
3. Single node, multiple GPUs: vLLM tensor parallelism
4. Multiple nodes: LeaderWorkerSet
5. The frontier: disaggregated prefill and decode
6. Exhaust the simple options first
7. Going deeper
8. Try it yourself

Training & Batch Workloads

1. The mental shift: services vs jobs
2. The Kubeflow Training Operator
3. Gang scheduling: all or nothing
4. Kueue: quota and queueing, the Kubernetes-native way
5. Checkpointing: the non-negotiable
6. Going deeper
7. Try it yourself

Storage & Data for AI Workloads

1. Three problems, three access patterns
2. Where should model weights live?
3. The pattern: download once, share everywhere
4. StorageClass essentials for AI
5. Measure what you bought
6. Shared memory: the invisible volume
7. GPUDirect Storage, in one paragraph
8. Going deeper
9. Try it yourself

Networking for AI: RDMA, NCCL, and the East–West Fabric

1. Why multi-GPU jobs are network-bound
2. The fabric choices
3. RDMA and GPUDirect RDMA, in plain terms
4. The NVIDIA Network Operator
5. NCCL: the layer that uses it all
6. Validate with nccl-tests
7. Going deeper
8. Try it yourself

Observability & Benchmarking

1. Three layers to watch
2. Layer 1: GPU hardware with DCGM
3. Layer 2: the serving engine
4. Layer 3: SLOs and goodput
5. A starter alert shortlist
6. Benchmarking with genai-perf
7. Going deeper
8. Try it yourself

Security, Multi-Tenancy & Cost Control

1. The multi-tenant GPU problem
2. Baseline: quotas and limits per namespace
3. Priorities: inference beats batch
4. Network isolation: default deny for model namespaces
5. Policy guardrails with Kyverno
6. Cost: idle GPUs are the fire
7. Going deeper
8. Try it yourself

Emerging Patterns & Next Steps

1. RAG on Kubernetes
2. Vector databases on Kubernetes
3. Agentic workloads
4. The Gateway API Inference Extension
5. Where the platform is heading
6. How to keep learning
7. Going deeper
8. Try it yourself

Appendix A: GPU Troubleshooting Field Guide

1. Symptom: Pod stuck Pending with “Insufficient nvidia.com/gpu”
2. Symptom: CUDA out of memory
3. Symptom: CrashLoopBackOff with a driver/library mismatch
4. Symptom: Device plugin not advertising GPUs
5. Symptom: NCCL hangs or timeouts in distributed jobs
6. Symptom: ImagePullBackOff on huge AI images
7. Quick index

Appendix B: Knowledge Quiz

1. Chapters 1–2 — Foundations & the GPU Operator
2. Chapter 3 — Scheduling & Sharing GPUs
3. Chapters 4–6 — Serving & Optimizing LLMs
4. Chapter 7 — Distributed Inference
5. Chapter 8 — Training & Batch Workloads
6. Chapters 9–10 — Storage & Networking for AI
7. Chapter 11 — Observability & Benchmarking
8. Chapter 12 — Security, Multi-Tenancy & Cost
9. Chapter 13 — Emerging Patterns
10. Answer key