Models Make Things Up

Your LLM demo worked. Then production happened: a provider outage turned into a minute-long spinner, the model invented a refund policy, a retry loop quadrupled the bill, and a user talked the assistant into a discount code.

Models Make Things Up is for backend engineers who have been asked to "add AI" to a real product. It treats the language model as what it is in production: a slow, billed, sometimes-down, occasionally wrong dependency, and shows the engineering that makes features built on it reliable.

What you will learn

• Treat model calls like any fallible dependency: latency budgets, deadlines, typed errors
• Turn model text into validated, typed data, and repair once instead of looping
• Version prompts like code, with the version on every request
• Retry the right errors with jitter and Retry-After; add circuit breakers and tested fallbacks
• Let models trigger actions without double refunds: idempotent tool calls
• Know the cost of every request, cache safely, and stop runaway spend with budgets
• Stream, queue or batch depending on who is waiting
• Build evals that catch regressions before users do
• Log enough to debug without leaking personal data
• Measure RAG retrieval separately from answers
• Contain prompt injection by limiting what the model can do, see and trigger
• Use AI near money safely: the model is an analyst, not a cashier
• Put agents on a leash: step, cost and time limits, escalation
• Roll out with shadow mode, flags and a kill switch, and design an LLM system in 45 minutes

What is inside: 14 chapters, 5 appendices (glossary, production-readiness checklist, LLM API error taxonomy, eval set template, sources), 9 diagrams, and an Interview Corner, Production checklist and TL;DR in every chapter. Every Go snippet comes from a companion repository (standard library only, every package tested). PDF (146 pages) and EPUB.

No model rankings, no hype. Provider-agnostic engineering that stays useful when models change.

Who it is for: backend engineers and tech leads building or reviewing LLM features, and engineers preparing for "design a system with an LLM" interviews.

Who it is not for: people looking for ML theory, fine-tuning, or a prompt-tricks collection.

Facts that can change were checked October 2026. All incidents are composites.