LLM system design for developers

Your LLM feature works on your laptop. Then real traffic shows up, the provider has a bad afternoon, the bill doubles, and someone pastes a prompt that talks your agent into touching things it shouldn't.

That gap, between a call that works and a system you'd trust in production, is what this book is about. It follows one assistant as it grows from a single team to a whole company, and each chapter fixes the next thing that breaks. You'll estimate load on the back of an envelope, budget latency and tokens, decide what's worth caching, add retries and fallbacks that don't make an outage worse, send easy requests to cheaper models, and treat structured output and tool calls as contracts. Later chapters put limits around agents, design against prompt injection instead of filtering for it, trace cost per feature, and wire evaluation into releases. Then comes the old question: hosted API, or run the model yourself?

Every chapter closes with a decision you could make on Monday, backed by measurement rather than defaults. The last chapter takes the system design questions AI engineering interviews tend to ask and answers them from the trade-offs.

The examples are in Python, and each one was run before publication. Where TypeScript does the job differently, there's a note. Claims about papers and libraries are checked against the original source. You should be at home writing code and know how a web service works. No machine learning background needed.