Leanpub Header

Skip to main content

Ferro: Build a Deep Learning Library from Scratch in Rust

Minimum price

$19.00

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
About

About

About the Book

Most of us use machine learning libraries we could not rebuild. This book builds one, from nothing, in Rust — no PyTorch, no libtorch, no BLAS, no Python at runtime — and checks every layer before standing on the next.

It starts with an array type: a buffer of numbers and a description of how to read it. Then the arithmetic on those arrays, then automatic differentiation checked against finite differences, then layers and optimizers that reach 98% on MNIST. Then speed: a matrix multiply within 1.03–1.38× of OpenBLAS, and a convolution stack that trains ResNet-18 to 93.26% on CIFAR-10. Then checkpoints that resume a run bit for bit. And finally reinforcement learning: tabular Q-learning that lands within 0.0196 of value iteration's exact answer, and a DQN agent that balances

CartPole on seeds nobody tuned it on.

Every phase ends in a gate that is a command, not a feeling, and the book is about how those gates are built as much as what they test:

  • An oracle wherever one exists — NumPy for the kernels, the definition of a derivative for autograd, value iteration for Q-learning.
  • Controls that must fail — runs that break one thing on purpose, so a passing test means something.
  • Predictions written down first — and reported as wrong when they were. The profiler that put un-threaded code at 6.8% of the work while the scaling curve said 27%. The Huber loss every DQN uses, which gives the wrong answer on a slippery grid.

The code in the book is the code in the repository. Every listing is extracted from the source by a script that fails if the two ever drift.

Who it is for: developers who can program and want to know what is inside a machine learning library. You do not need prior Rust expertise or ML background beyond the headlines; the book supplies it, and includes a vocabulary appendix.

In progress: this edition covers Phases 0–9 of the plan, from the empty workspace to deep reinforcement learning. Later phases — the DQN family, policy gradients, tree search, and AlphaZero learning Connect 4 — will be added as they are built, and every update is free to existing readers.

Share this book

Author

About the Author

Contents

Table of Contents

Preface

  1. Why build it at all
  2. Gates that are commands
  3. How the claims are checked
  4. Who this book is for
  5. About the code
  6. About the numbers
  7. What this volume covers
  8. Foundations

Before Any Code

  1. The rules of the build
  2. Fourteen crates, all empty
  3. The oracle
  4. Reading NumPy’s file format
  5. An error type before a tensor
  6. One command for the gate
  7. Every check on, before there is anything to check
  8. What the gate measured
  9. Phase 1’s gate, written in Phase 0
  10. What Phase 1 inherits

A Tensor Is a Shape, Some Strides, and a Pointer

  1. Six fields, one of which holds numbers
  2. Where the numbers live
  3. Lying about the strides
  4. The contiguity fast path
  5. Testing arithmetic nobody can enumerate
  6. One kernel, six types
  7. The two operations that decide whether an agent learns
  8. Owning the random numbers
  9. The gate
  10. What the tensor does not do yet
  11. Arithmetic and Derivatives

Kernels That Are Obviously Correct

  1. One thread, on purpose
  2. The machinery every kernel shares
  3. Formulas that are correct and still wrong
  4. An error function from 1969
  5. Two tensors at once
  6. Why a + b does not compile
  7. Writing in place
  8. The slowest matrix multiply I could defend
  9. What the baseline says
  10. Testing without a second implementation
  11. What Phase 2 leaves for autograd

Derivatives You Can Check

  1. The checker comes first
  2. The tape is an array
  3. The chain rule is a sum
  4. Broadcasting, backwards
  5. Derivatives that do not exist
  6. Where activation memory goes
  7. What the phase shipped
  8. What Chapter 5 inherits
  9. Learning

Layers, Optimizers, and Ninety-Eight Percent

  1. The gate, and why it is a number
  2. A parameter that never trains
  3. A parameter is not a tensor
  4. The mode is an argument
  5. What an optimizer does
  6. Cross-entropy is two functions fused
  7. Ninety-eight percent
  8. The golden run
  9. The cheap tests that name the cause
  10. What the phase leaves for the next one
  11. Speed

A Matrix Multiply Within Reach of OpenBLAS

  1. The reference that was the wrong reference
  2. Every stage stays in the tree
  3. Sixteen times from swapping two loops
  4. The stage that made it slower
  5. Packing: the copy that pays for itself
  6. Lane selection instead of broadcasting
  7. Miri and the new unsafe
  8. Threading is a cache decision
  9. The pool nobody needed
  10. Against the gate
  11. What the phase leaves for Chapter 7

Convolutions, and Ninety-Three Percent on CIFAR-10

  1. What a convolution does
  2. A convolution is a matrix multiply
  3. Going backwards, the copies add
  4. Normalization, and the conventions that cost twenty points
  5. ResNet-18 for a 32-pixel image
  6. Sixteen seconds a step
  7. Three elementwise functions
  8. Finishing the first push
  9. The profiler that counted threads
  10. Welford, threaded rather than replaced
  11. Packing cost more than multiplying
  12. A refusal is not a fallback
  13. Two things I costed instead of building
  14. Where it stopped
  15. Ninety-three point two six
  16. What the gate does not prove
  17. What the next phase inherits
  18. Plumbing

A Resume That Is Bit-Identical

  1. Why the gate asks for zero
  2. A loader whose output ignores its threads
  3. Samplers, and checking a bound once
  4. A file format that someone else can read
  5. What a checkpoint has to hold
  6. Making a passing gate mean something
  7. The gate failed
  8. The state that wasn’t a parameter
  9. Moving the state instead of extending the machinery
  10. A log that survives the crash
  11. What the gate does and does not prove
  12. What this leaves for Chapter 9
  13. Reinforcement Learning

The One Place With an Exact Answer

  1. Reinforcement learning from nothing
  2. Two traits, on purpose
  3. Two booleans
  4. GridWorld
  5. The exact answer
  6. Learning from experience
  7. The gate
  8. A random agent, and an identity instead of a range
  9. Turning a trajectory into a target
  10. Classic control, matched to Gymnasium
  11. Wrappers, and the clip that could never fire
  12. The parallel environments that made it slower
  13. The same table, balancing a pole
  14. What the table cannot do

DQN, and the End of Exact Answers

  1. From a table to a network
  2. The buffer
  3. The target network, and the clone that is not a copy
  4. Exploration as a value
  5. The skeleton every later algorithm plugs into
  6. The agent
  7. The two checks the plan asked for
  8. The first run passes by a tenth
  9. A result below random
  10. An exact answer after all
  11. The policy that does not stay learned
  12. The gate
  13. Two controls, and one of them does not fail
  14. An impossible number, and the next phase

Appendix A: Vocabulary

  1. What the thing is trying to do
  2. The data structure
  3. Making it learn
  4. The training loop
  5. Whether it worked
  6. The layers
  7. The datasets and the gates
  8. The performance words
  9. Deep reinforcement learning

Appendix B: Reproducing the Results

  1. Everything CI runs
  2. Chapters 1 to 4: the tensor, the kernels and autograd
  3. Chapter 5: MNIST
  4. Chapter 7: CIFAR-10
  5. Chapter 8: checkpoints
  6. Chapter 9: environments and tabular RL
  7. Chapter 10: DQN
  8. The book’s own check
  9. A note on hardware

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub