The AI Gateway Playbook

Every organization that gets serious about LLMs eventually hits the same wall. API keys get hardcoded on laptops, extra keys appear because the first one got rate-limited, and months later the platform team discovers that the model bill has outgrown the team that owns it and that nobody can answer "who is spending what, on which model, for what feature?"

The AI Gateway Playbook is a practitioner's guide to the system that fixes that: an in-house LLM gateway. It is a thin, boring, auditable layer of infrastructure that sits between your codebase and every model provider, and treats model access the way any mature platform treats secrets, budget, and access control.

You will learn how to build:

• A model registry that syncs models, availability, and pricing across multiple providers, so your code talks to stable aliases instead of vendor model IDs
• A unified API surface with one wire protocol for chat, streaming, embeddings, and tool-calling
• A keyless auth layer where no static LLM API key is ever issued to a human, using managed identities, on-behalf-of flows, and a six-tier RBAC model with per-role rate limits and daily token quotas
• A metering and cost pipeline: token accounting per model, tenant, and user; budgets with soft and hard limits; dashboards and alerting; quota-increase approvals; and monthly reconciliation against provider invoices
• An MCP server that exposes internal SaaS and office tools as Model Context Protocol tools at 90+ tool scale, secured with OAuth 2.1 + PKCE, envelope-encrypted secrets, PII masking, and a self-service portal
• A RAG-based team assistant with a worked CI/CD failure-analysis example and an evaluation harness that keeps it honest
• The operations layer: audit retention, key rotation without static keys, observability, incident runbooks, and deployment and migration checklists

Who this book is for: senior infrastructure and platform engineers, AI/ML platform engineers, DevOps engineers and SREs who will run the system, and tech leads or engineering managers who need to make and defend the build-vs-buy decision. The book is deliberately cloud-generic: components are described by role, so you can map the patterns onto your own stack.

What you get: about 32,000 words across eight chapters, plus front matter and an index. It is practitioner-oriented throughout, with architecture and sequence diagrams, YAML/JSON configs and code sketches, decision tables, a pitfalls section at the end of every chapter, and a full production-readiness checklist.

How this book was made: drafted with AI assistance from the author's hands-on experience designing and running an in-house LLM gateway; all examples are generic and anonymized.