Enterprise AI Platform

Somewhere there is a server with two GPUs for which no operational process exists.

Developers have long been using ChatGPT and Claude, often through private accounts. Business units have built proofs of concept that nobody operates. The data protection officer has questions. Procurement has seen an invoice that nobody can attribute to a team.

Whoever is to turn that into an operable platform needs different knowledge from the data scientist and different knowledge from the application developer: GPUs in the cluster, the mechanics of inference servers, access control on models, cost attribution — and the question of which data may reach which model.

HANDS-ON: 23 LABS

The book builds the platform piece by piece on a reference lab you can rebuild: one NVIDIA GPU with 24 GiB, passed through to a virtual machine, RKE2 across several nodes. Every lab names its prerequisite and its expected output — so you can tell whether it worked, and how you notice when it did not.

One lab needs more: the tensor parallelism measurement in Chapter 9 rents a two-GPU instance by the hour, one to three euros for the whole series. Where the hardware runs out, the book says so instead of pretending.

WHAT YOU OPERATE IN THE LABS

vLLM — batching, prefix caching, chunked prefill, preemption and the KV cache made visible through the scheduler's own metrics; engine arguments set from a calculation instead of by guessing.

KServe — ServingRuntime and InferenceService, declarative model serving, canary rollouts, autoscaling on the queue, scale-to-zero.

LiteLLM as the AI gateway — virtual keys, budgets, rate limits, model allowlists bound to the data class, fallbacks, spend logs per team.

NVIDIA GPU Operator, driver, Container Toolkit, DCGM Exporter — from PCI passthrough to a GPU the scheduler can allocate; MIG and time-slicing, including the memory isolation test that shows why one of them is unsuitable for tenants.

Keycloak and OpenBao — OIDC claims down to team and data class, and provider keys that no developer ever sees.

Prometheus and Grafana — the metrics that carry meaning for inference, and the ones that mislead.

ArgoCD and Harbor — moving a 20 GB model artefact through a GitOps chain, with a rollback that takes minutes.

pgvector — a retrieval path with permission filtering inside the query, and erasure you can demonstrate.

Also covered: Ray, Qdrant, Milvus, PostgreSQL, Redis, External Secrets, Proxmox VE, RKE2.

NOT A TUTORIAL

The labs are there to make things measurable, not to be followed once and used up. The aim is not to explain tools — tools age. The aim is to make the derivations available: why the KV cache and not the weights is the capacity bottleneck, why 100 percent GPU utilization can mean nothing at all, why self-hosting pays off above a certain utilization and not above a certain number of users.

Every chapter carries a version stamp naming what it was written and checked against. Chapters 2 to 16 each end with a troubleshooting table; every chapter ends with interview questions at senior level, with what separates a weak answer from a convincing one.

From the contents: I LLM Inference Fundamentals · II GPU Infrastructure · III Model Serving · IV The Enterprise Layer · V Operations · VI Synthesis

Assumes Kubernetes, GitOps, identity providers and monitoring in production. This is not a Kubernetes book and not a machine learning book — it starts where those leave off.

386 pages · 17 chapters · 4 appendices · 23 labs · PDF and EPUB