Leanpub Header

Skip to main content

Distributed Systems from the Ground Up: Consensus, Storage Engines, Traffic Control, and Fault-Tolerant Architecture

Minimum price

Free!

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
About

About

About the Book

Distributed systems engineering operates under physical and theoretical constraints that single-node abstractions conceal: asynchronous message delivery, unbounded network delay, independent component crashes, clock drift across unsynchronized oscillators, and volatile kernel page caches. Designing reliable infrastructure requires deriving architectures directly from explicit safety and liveness invariants, fault boundaries, and quantitative resource limits rather than informal component sketches.

In this book, I examine the architecture of fault-tolerant distributed systems from first principles. For every system, I establish the actors, verifiable functional contracts, explicit latency and availability Service Level Objectives (SLOs), and storage, compute, and bandwidth envelopes prior to introducing algorithms, wire protocols, or deployment topologies.

Organization of the Text

The twenty-eight chapters are grouped into six thematic parts:

  • Part I (Chapters 1 to 6): Distributed Key-Value Store. In Chapter 1, I specify the system architecture, consistency model, partitioning scheme, and failure-handling contracts of a strongly consistent key-value store. In Chapters 2 to 6, I detail the low-level design and empirical verification of its core mechanisms: durable write-ahead logging, leader-follower quorum replication, Raft leader election with monotonic term fencing, consistent-hash request routing, and group commit.
  • Part II (Chapters 7 to 13): Core Infrastructure Primitives. I analyze foundational platform services that enforce fairness, ordering, caching, coordination, and asynchronous decoupling: distributed rate limiting, unique identifier generation, distributed caching, distributed locking, partitioned message queues, job scheduling, and notification delivery.
  • Part III (Chapters 14 to 18): Content, Media, and Communication Systems. I analyze read-intensive and media-delivery architectures: URL shortening, social news feeds, chunked object storage, adaptive video streaming, and real-time messaging.
  • Part IV (Chapters 19 to 21): Transactional and Marketplace Systems. I analyze domains governed by strict transactional integrity, concurrency control, and real-time spatial state: payment processing with double-entry ledgers, high-contention ticket booking, and geospatial ride-hailing dispatch.
  • Part V (Chapters 22 to 24): Search, Indexing, and Observability. I analyze low-latency prefix indexing, polite distributed web crawling, and high-cardinality metrics ingestion and monitoring.
  • Part VI (Chapters 25 to 28): Synchronization and Low-Latency Streaming. I analyze conflict resolution, operational convergence, deterministic execution, and stream processing: file synchronization, collaborative editing, a single-threaded stock exchange matching engine, and click aggregation.

Architectural Methodology

Across every system design chapter, I adhere to an explicit engineering progression aligned with formal systems documentation standards (ISO/IEC/IEEE 26514):

  1. Requirements and Capacity Bounds: I specify actors, use-case boundaries, numbered functional and non-functional requirements, and order-of-magnitude storage, throughput, and memory calculations upfront.
  2. External and Internal Interface Contracts: I define HTTP/2 or gRPC endpoints, preconditions, idempotency semantics, error payloads, and storage schemas before examining internal optimizations.
  3. Control-Plane and Data-Plane Separation: I isolate low-frequency metadata management, topology coordination, and policy compilation from the latency-critical request path.
  4. Failure Modes, Tail Latency, and Observability: I evaluate network partitions, hot-key skew, lock contention, and gray failures, specifying automated recovery invariants and Service Level Indicators (SLIs) for each failure domain.

Companion Implementations and Prerequisites

The companion code/ directory contains the runnable Part I storage subsystems and the workload simulations cited across Parts II to VI, allowing readers to reproduce the throughput measurements, skew distributions, and crash-recovery experiments reported in the text.

For the system design chapters, I assume foundational knowledge of HTTP semantics, relational databases, data structures, and TCP/IP networking. For the low-level implementation chapters in Part I, I additionally assume reading familiarity with Rust. Every distributed systems primitive is derived when first introduced and consolidated in each chapter’s engineering glossary.

References and Provenance

The protocols and trade-offs presented in this volume draw upon primary distributed systems literature and published production engineering reports, including work on Raft consensus, Dynamo, Spanner, CockroachDB, Stripe’s atomic-phase idempotency and GCRA rate-limiting systems, and classical transaction-processing theory. Full bibliographic citations appear at the close of each chapter.

Arpan Pathak

Share this book

License

Creative Commons Attribution 4.0 International License

Author

About the Author

Arpan Pathak

I am a software engineer with over 8 years of experience building and operating cloud infrastructure, storage systems, payment gateways, and AI serving platforms at Microsoft, Oracle, Amazon, and Razorpay. My work focuses on distributed systems, systems programming, and parallel GPU computing. I wrote this book to present distributed system architectures rigorously, deriving each design from explicit requirements, failure models, and executable verification.


Website: https://arpanpathak.github.io

GitHub: https://github.com/arpanpathak

Get the free Community Edition

You can get the free Community Edition in PDF or EPUB just by sharing your name and email address with the author.

 

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub