100 LLM Autopsies

A model that crashes is a good day.

Crashes have stack traces. They have a line number, a repro, an owner, and a fix that either works or does not. Nobody debates for three weeks whether a segmentation fault is real.

The failures in this book do not crash. They return an answer — fluent, well-formed, confident, and wrong by four percent. They pass the test suite. They ship. They sit in production for months while a team optimises around them, and they are found eventually by accident, or by an outsider, or by someone who finally opened an artifact that had been sitting there since the first deployment.
This book was created through a process that combines careful human planning, content direction, and advanced AI technology, followed by thorough refinement and review to ensure a high-quality final work.

This is a book of one hundred such failures, opened up and examined.

Not a book about models being unreliable

Of the hundred cases here, the number caused by the model's weights being wrong is very small. The rest live in the machinery around it: a tokenizer, a template, a mask, a cache key, a config default, a retry, a metric. Those are ordinary software components, and they fail in ordinary software ways.

What makes them hard is that the system's output stays plausible while they fail — so nothing alerts, nothing throws, and the only signal is a number that is slightly worse than it should be.

• A duplicated beginning-of-sequence token that cost seven points of instruction adherence.
• An evaluation whose score changed by four points depending on the order the questions were asked in.
• A prefix cache bounded by entry count, and therefore unbounded in memory.
• A safety mask that, when retrieval returned nothing, made the model answer from a uniform blend of everything.
• Two identical support tickets, classified differently, every time, correctly.

Four levels, defined by how much you must hold in your head

The cases are not organised by topic. They are organised by how much of the system you must understand at once to find the bug.

• Level I — Beginner (01–30). One model, one machine, one request. Every bug is visible in an artifact somebody could have opened on the first day: a token array, a request body, a startup log, a cache key.
• Level II — Intermediate (31–60). The bug does not exist for a single request. It is created by grouping, batching, adapting, indexing or training. You now need to reproduce a situation, not an input.
• Level III — Advanced (61–85). Load, time, kernels, replicas. These bugs do not exist on your laptop, and several do not exist until the system has been running for six hours.
• Level IV — Expert (86–100). The deceptive cases. The evidence is complete, the reasoning is sound, and the conclusion is wrong.

Every case follows the same rhythm

Case card, report, evidence, a failed suspect or two, the turn, the verdict, the fix — and where relevant, what the fix broke. Then the lesson, generalised past the specific bug, and a diagnostic instrument you keep.

The failed suspects are not padding. Learning which plausible explanations to discard is most of the skill, and a book that shows only correct reasoning teaches nothing about how to reason before you know the answer.

What you actually keep

If you remember a hundred bugs, you have learned a hundred bugs, and the hundred and first will be new.

So the cases are not the payload. Running through them is a set of 63 instruments — small, cheap, reusable diagnostic procedures, each introduced inside the case that makes you want it. Most take under twenty minutes to build; several are a single assertion. They share one property:

Every one of them answers a question about what is actually happening, rather than what the code says should be happening.

On evidence, and on trust

Every case card carries an Evidence basis line, with one of four values — documented incident, documented behaviour, reconstructed, or composite — because you are entitled to know how much of what you are reading actually happened. Most cases are reconstructed, and the book says so on the page rather than implying a hundred first-hand war stories.

What is not reconstructed is the mechanism. Every root cause is a real failure mode of real software. Where a case reports a measurement, the arithmetic in the table reconciles. Where a case gives you code as a fix, the code runs.

Who this is for

Anyone who has shipped a system with a language model inside it and has had the experience of not being able to explain what it just did. You do not need to have trained a model. You do need to be comfortable reading code — Python where models, training and evaluation live; C++ and configuration where engines and serving live.

What it is not

It is not about prompt engineering. It is not an API reference. And it is not a book about models being unreliable — almost every model in these hundred cases behaved exactly as it was built to behave. It was given two beginning-of-sequence tokens, or a mask that let it attend to padding, or a prompt in a format it had never been trained on, and it did the only thing it could do with what it was handed.

The model is rarely the patient. It is usually the witness.