Leanpub Header

Skip to main content
Go to Leanpub.com

Beyond Uptime: Reliability Engineering for AI Infrastructure

Beyond Uptime: Reliability Engineering for AI Infrastructure
This book is 90% completeLast updated on 2026-09-27
Sreejith Kaimal

When a 2,048-GPU training cluster running a 400-billion-parameter model sees its iteration time double without triggering a single alert, firing a CPU panic, or dropping a single packet, you don't have a machine health problem : you have a coordination failure.

Beyond Uptime is the definitive field guide for Site Reliability Engineers, platform leads, and cloud architects tasked with keeping frontier AI compute running at peak efficiency. Moving beyond deceptive "GPU Utilization" charts, this book uncovers the hidden mechanics of barrier synchronization, lossless network congestion cascades (PFC/ECN), silent PCIe downtraining, and Silent Data Corruption (SDC). Packed with eBPF tracing methods, streaming gNMI telemetry rules, statistical tail analytics (MAD/CUSUM), and dynamic circuit-breaker strategies, Beyond Uptime gives you the battle-tested framework needed to isolate stragglers in minutes rather than days. Stop paying the silent tax on your AI infrastructure!

Free With Membership

With Membership

Free!

$25.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
About

About

About the Book

Beyond Uptime: Reliability Engineering for AI Infrastructure bridges the gap between high-level distributed systems thinking and low-level physical telemetry. Anchored by a realistic $4.2M forensic case study (Operation Slowburn), this book provides a complete, battle-tested framework for diagnosing, preventing, and containing hidden straggler failures across the entire AI compute stack.

Modern AI training relies on mathematical barrier synchronization (such as NCCL AllReduce collectives), the throughput of an entire multi-million-dollar cluster collapses to the speed of its single slowest link or participant—the "straggler". Standard site reliability engineering (SRE) tools, built to monitor independent, loosely coupled web services, evaluate whether single nodes are online, not whether the collective is in sync. As a result, subtle physical issues like PCIe link downtraining, HBM memory degradation, and Priority Flow Control (PFC) network pause cascades regularly burn millions of dollars in silent stalls without triggering a single alert, firing a CPU panic, or dropping a packet.

Share this book

Categories

Site Reliability EngineeringDigital ForensicsNetworking

Feedback

Email the Author

Author

About the Author

Sreejith Kaimal

Sreejith Kaimal is a Principal Site Reliability Engineer and AI Infrastructure Architect with over 25 years of experience spanning the enterprise software development lifecycle. Currently at C3.ai, he specializes in high-availability AI platforms, cloud-native solutions, AIOps, and advanced site reliability engineering across hybrid cloud environments. His extensive career includes leadership roles at organizations such as OpenCrowd, Prudential Financial, UnitedHealth Group, and Cognizant, where he has driven large-scale digital transformation, DevOps modernization, and fault-tolerant architecture designs. Sreejith holds advanced degrees in Information Systems and Electronics and is recognized for his thought leadership in distributed compute reliability, physical-layer diagnostics, and multi-node cluster coordination.

Contents

Table of Contents

Beyond Uptime : Reliability Engineering for AI Infrastructure

  1. Disclaimer & Limitation of Liability

How AI Infrastructure Actually Works

  1. What is an AI Model
  2. Why this requires GPU’s rather than CPU’s
  3. Visualizing the AI Factory Floor
  4. How Tiny Delays Compound in AI
  5. The Modern AI Stack : A Glossary Through the Same Factory Lens
  6. Beyond Network Rules : A Systems-Thinking Argument

Operation Slowburn

  1. How can a system be healthy and still be failing?

The Synchronization Problem

  1. An Intuitive Model: Barrier Synchronization
  2. From the Table to the Training Run
  3. Why this is unavoidable
  4. A Worked Capacity-Planning Illustration

Operation Slowburn : A Forensic Reconstruction

  1. Incident Premise and Cluster Configuration
  2. The Forensic Investigation

Specialized Networking Technologies

  1. Limits of Ordinary Networking
  2. RDMA, RoCE, and InfiniBand
  3. A Cautionary Case: Fast Transport Is Not Sufficient Alone
  4. Comparing the Three Fabrics Along the Dimensions That Matter
  5. Transport Reliability Semantics: What RDMA Guarantees and What It Does Not

Lossless Fabric Bottlenecks

  1. The Congested Highway
  2. Priority Flow Control and Cascading Pause Frames
  3. Hash Polarization: A Second Congestion Mechanism
  4. The Diagnostic Implication

The Anatomy of Silent Data Corruption

  1. Transient Voltage Fluctuations & Thermal Noise
  2. Cosmic Radiation & Single-Event Upsets (SEUs)
  3. Manufacturing Variance & “Mercurial Cores”
  4. Passive Hardware Telemetry Checklist
  5. Memory Subsystem Degradation
  6. The Structural Argument

Systematic RCA - The Failure Graph

  1. The Fleet Aggregation Paradox
  2. Observability as Evidence Collection
  3. Systematic , Layer-by-Layer Method
  4. A Divergent Presentation of the Same Failure Class
  5. The Failure Graph as a Formal Structure

Generalizing the Incident :A Practitioner Framework

  1. Why Generalization Is Possible at All
  2. From Forensics to Foresight
  3. Toward Automated Detection
  4. A Concrete Detection Formalism: From Fleet Averages to Per-Rank Tail Statistics
  5. Preventive Engineering Practices
  6. Organizational Practices: Building the Team Around the Method

AI Investigating AI : Towards Autonomous Infrastructure

  1. Machine Learning as the Investigator, Not Only the Instrument
  2. Toward Autonomous AI Infrastructure
  3. An example of a Harnessed Agent Cluster running an AI model
  4. An Agenda for Further Research
  5. References

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

Download Sample PDFDownload Sample EPUB

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub