How AI Infrastructure Actually Works

An AI model is not a reasoning mind; instead, it is an incredibly large pattern-finding system. It is a computer program trained against millions or billions of examples, words, images, sounds, or videos, from which it does not memorize individual examples but instead extracts statistical regularities, mathematical relationships between inputs and the outputs those inputs tend to produce. When the trained model is later given a new prompt, it draws on those extracted regularities to predict the most probable continuation, not by consulting anything resembling stored facts, but by computing a probability distribution over possible next outputs and selecting from it. A model does not think in the sense a human does; it performs, at a very large scale, a very sophisticated form of statistical prediction. Picture this mechanism concretely as an enormous equalizer board, the kind used in audio mixing, except instead of a few dozen sliders, this board has billions or trillions of them, each called a parameter or a weight.

During training, the model is shown an example and produces a guess, for instance, a predicted next word in a sentence, and is told whether that guess was correct. When it is wrong, a procedure called back propagation nudges a very large number of these sliders by a small amount in the direction that would have made the correct answer slightly more probable next time¹ . This nudging repeats across billions of examples until the sliders settle into a configuration that produces good predictions consistently. Once training concludes, those sliders are frozen; this frozen state is what practitioners mean when they refer to a model’s weights. At inference time, a prompt is converted into numerical vectors, passed through this enormous fixed network of sliders, and converted back into a probability distribution over the most likely next output, a process that, unlike training, does not modify any of the sliders themselves.

Later in this chapter, the analogy of a huge, fast factory assembly line comes up again and again, and it’s worth stating it explicitly and completely here in order to make every subsequent discussion on synchronization, on congestion, and on silent hardware degradation much easier to follow.

The CPU will now act as the factory manager. It is smart and very flexible, but it cannot do the manufacturing work itself; it is only used to make a decision about the logic, the flow of work, and what to do next.

The GPU acts as the workforce of a factory where each worker is simply doing the same part of the job, but they are capable, as a group, of doing this at vast numbers and extremely high efficiency, in this case, a simple arithmetic operation.

The GPU’s built-in memory (VRAM) acts as the workbench directly next to each worker. Any raw material put on this workbench is instantly available to the worker; if the workbench is too small to work on, the worker’s choice is to either walk to a distant warehouse or ordinary storage to retrieve more material or to walk back to the workbench. This is exactly how the Out of Memory error occurs when a training job’s batch size (amount of material loaded onto the workbench at one time) is larger than the physical capacity of the workbench, and a practitioner reduces the batch size or adds GPUs with larger VRAMs, a larger workbench.

Networking is like a logistical freeway that links several small factory structures that are spread out around the country, with each building being a single physical server node with a lot of GPUs in it. The significance of this role is enormous to the argument that follows in the rest of this chapter: even if each manager and every worker in each individual factory building were working at perfect efficiency, the moment the trucks carrying intermediate parts between factories got stuck in the traffic jams on the road the entire multi-factory production line would come to a standstill, regardless of how each building was doing on its own. This same intuition is captured in Section 3, albeit in a much more explicit manner, in the form of barrier synchronization.

This same factory language applied to the data path used to move one batch of training data from disk to a GPU’s workbench, breaks the process into three specific steps, and these steps are each named explicitly because they will be encountered in the following text, under their appropriate technical names.

1. Data Loading: the manager of the factory, the CPU, gets raw materials from the warehouse. The training framework makes a read request to the operating system, asking it to return the addresses in the physical storage where the data is located, and the CPU puts the request on a list and passes it to the storage drive’s own hardware controller, which instructs it to start performing the transfer.
2. Data staging: The material is in the storage drive’s own controller, and with a method known as Direct Memory Access, it is moved directly to the workbench (or host system memory) and only notifies the CPU when the transfer is finished, while the factory manager is not involved in the transfer process and can attend to other work.
3. Data Processing: processing of the delivered material, the mass processing, is performed by the assembly line laborers (the GPU), where the parameters of the model would be updated as outlined in Section 1.1, based on the thousands of delivered units. Not just a delay inside the GPU, but anywhere of the three delays above can spell trouble for a whole fleet of healthy workers, as the worked example that follows in Section 3.2 shows when training is CPU-bound.