Tokens per Dollar

Running open-weights models like Llama 3.1 in production is expensive in a way almost nothing else in your cluster is. An idle nginx pod wastes pennies; an idle 2×A100 model server wastes $10 an hour. Most teams are paying full price for GPUs that do useful work 10–30% of the time — same old overprovisioning habits, at 500x the cost.

This micro-book is the fix, written for platform engineers, SREs, and cloud architects who know Kubernetes but not inference internals. No theory tour, no framework survey — just the levers that move the bill, in order of payoff:

• Think in cost per million tokens, the only unit that matters — and measure yours before touching anything
• Tune vLLM properly: the three flags worth 2–4x throughput on identical hardware, with a full production Deployment manifest
• Use quantization (FP8, INT4/AWQ) to halve your GPU count — plus the golden-set eval workflow that makes it safe
• Turn on prefix caching and sticky routing to stop paying for the same system prompt thousands of times an hour
• Autoscale on real token traffic with KEDA and Karpenter — queue depth, not the CPU and "GPU utilization" metrics that lie
• Buy compute well: spot burst tiers, MIG, and why commitments come last, not first

It closes with a realistic case study that takes a $29,200/month 70B deployment to $8,600 — a 71% cut, with better peak latency — and a one-page checklist of the whole book.

Every example is working YAML. Every claim comes with the arithmetic. Deliberately short: read it in an evening, apply it this sprint, and it pays for itself the first day.

What Readers Will Learn (bullets field)

• How to compute and track cost per million tokens for any deployment
• The VRAM budget model (weights + KV cache) that determines your real concurrency ceiling
• Production vLLM configuration: --max-model-len, --gpu-memory-utilization, --max-num-seqs, probes, and model-weight loading that doesn't wreck cold starts
• When FP8 vs INT4 quantization is safe, and how to verify quality with your own eval set
• Prefix caching, sticky routing, and response caching — and which one can silently return wrong answers
• KEDA autoscaling on vllm:num_requests_waiting with asymmetric scale-up/scale-down, plus Karpenter node provisioning with a blast-radius cap
• Spot-instance burst tiers, MIG partitioning, and benchmark-driven GPU selection

Who This Book Is For

Platform engineers, SREs, DevOps engineers, and cloud architects running (or about to run) open-weights LLMs on Kubernetes. Assumes working Kubernetes knowledge; assumes zero inference knowledge. Not for you if you need training/fine-tuning guidance or a managed-API cost guide.