AI, Bottom Up: From the First Token to the Frontiers of Intelligence.

Through story, mechanism, and derivation, the book reveals how language becomes something a machine can transform, how coherent responses emerge one token at a time, and where fluency ends and truth, memory, and context must begin.

Once, a thought lived only as long as someone could remember it.

Then memory left the mind.

It entered marks, clay tablets, manuscripts, printing presses, and digital text. A story could outlive its speaker. An instruction could cross a river. Knowledge could travel farther than any traveler.

Centuries later, a machine meets language.

But the machine does not see a river, hear a story, or recognize a traveler. It receives encoded marks—Unicode, bytes, fragments, and arbitrary integer IDs. Somewhere between the sentence we understand and the numbers a machine can process, language crosses a hidden boundary.

Token opens that boundary.

Consider one unfinished sentence:

The river carried stories farther than any _____.

You may already be thinking traveler. How does a language model arrive at a word that fits?

During training, it acquires broad patterns from vast amounts of text and distributes them across its learned weights. During generation, it combines those global patterns with the local context of the sentence. It scores possible next tokens, selects one, adds it to the sequence, and asks again:

What comes next?

Each answer changes the next question. One context-sensitive choice follows another until tokens become a sentence that appears coherent, meaningful, and intelligent.

The book follows this transformation one hidden door at a time. The alphabet becomes encoded text. Text breaks into tokens. Token IDs find locations in learned geometry. Order enters the sequence. Attention connects each position to what came before. Transformer layers reshape the representation. Training leaves patterns in the weights. Logits open a horizon of possible continuations. A decoding policy chooses one—and the journey begins again.

Most books introduce AI after the machinery is already running. Token begins earlier, when human expression first becomes something a machine can transform. Stories invite the general reader forward; Mechanism boxes reveal the construction; Derivations and experiments let technical readers verify each step; and “Where the Analogy Breaks” sections prevent a beautiful explanation from becoming a misleading one.

By the end, the black box has become a sequence of inspectable transformations. You will understand not only how a token participates in producing language, but also what tokens and learned weights cannot provide by themselves: enduring memory, evidence, provenance, purpose, and the wider world surrounding a sentence.

That boundary leads to the next question—and the next volume:

A token has entered the machine. But what context should travel with it?

“Leave it by the bank before noon.”

Every word is familiar. Yet the instruction is incomplete. Which bank? Leave what? Whose noon? Who made the request—and do they have the authority to do so?

This gap between readable language and usable meaning is context.

Volume I, Token, followed language into the machine and showed how words become numerical representations that a model can process. But a token cannot determine its own meaning. It arrives surrounded by possibilities. Context: Drive the Known to Derive the Unknown begins at that boundary and asks: what selects the information that makes one interpretation appropriate now?

Following one deceptively simple instruction, the book travels through context windows, retrieval, embeddings, reranking, RAG, conversation state, persistent memory, knowledge graphs, MCP, provenance, security, and evaluation. It reveals that context is not merely a larger prompt or a pile of retrieved documents. It is an engineered chain of decisions: what enters, what is excluded, which source is trusted, what is transformed, and what may be lost.

Stories make each problem intuitive. Mechanisms reveal the system underneath. Derivations and laboratories make the ideas inspectable and reproducible. A recurring Canonical Context Trace connects the journey, showing how an ambiguous request becomes a bounded, authorized, and traceable context packet.

But context has a limit. It can find relevant evidence, preserve conflicting sources, and expose uncertainty. It cannot, by itself, determine what validly follows from that evidence. Relevance is not reasoning, and a citation is not a conclusion.

That unresolved boundary opens Volume III, Reasoning. If Token asks how language enters the machine, and Context asks what information should surround it, Reasoning asks the next unavoidable question:

Given what is known, what can the system justifiably conclude?