Beyond Uptime: Reliability Engineering for AI Infrastructure
/
Disclaimer & Limitation of Liability
Disclaimer & Limitation of Liability
https://leanpub.com/forensic-ai
Original Research & Independent Works
https://leanpub.com/forensic-ai
Reader Discretion & Assumption of Risk
https://leanpub.com/forensic-ai
No Warranties & Exclusion of Liability
https://leanpub.com/forensic-ai
Up next
How AI Infrastructure Actually Works
In this part
How AI Infrastructure Actually Works
What is an AI Model
Why this requires GPU’s rather than CPU’s
Visualizing the AI Factory Floor
How Tiny Delays Compound in AI
The Modern AI Stack : A Glossary Through the Same Factory Lens
Beyond Network Rules : A Systems-Thinking Argument
Operation Slowburn
How can a system be healthy and still be failing?
The Synchronization Problem
An Intuitive Model: Barrier Synchronization
From the Table to the Training Run
Why this is unavoidable
A Worked Capacity-Planning Illustration
Operation Slowburn : A Forensic Reconstruction
Incident Premise and Cluster Configuration
The Forensic Investigation
Specialized Networking Technologies
Limits of Ordinary Networking
RDMA, RoCE, and InfiniBand
A Cautionary Case: Fast Transport Is Not Sufficient Alone
Comparing the Three Fabrics Along the Dimensions That Matter
Transport Reliability Semantics: What RDMA Guarantees and What It Does Not
Lossless Fabric Bottlenecks
The Congested Highway
Priority Flow Control and Cascading Pause Frames
Hash Polarization: A Second Congestion Mechanism
The Diagnostic Implication
The Anatomy of Silent Data Corruption
Transient Voltage Fluctuations & Thermal Noise
Cosmic Radiation & Single-Event Upsets (SEUs)
Manufacturing Variance & “Mercurial Cores”
Passive Hardware Telemetry Checklist
Memory Subsystem Degradation
The Structural Argument
Systematic RCA - The Failure Graph
The Fleet Aggregation Paradox
Observability as Evidence Collection
Systematic , Layer-by-Layer Method
A Divergent Presentation of the Same Failure Class
The Failure Graph as a Formal Structure
Generalizing the Incident :A Practitioner Framework
Why Generalization Is Possible at All
From Forensics to Foresight
Toward Automated Detection
A Concrete Detection Formalism: From Fleet Averages to Per-Rank Tail Statistics
Preventive Engineering Practices
Organizational Practices: Building the Team Around the Method
AI Investigating AI : Towards Autonomous Infrastructure
Machine Learning as the Investigator, Not Only the Instrument
Toward Autonomous AI Infrastructure
An example of a Harnessed Agent Cluster running an AI model
An Agenda for Further Research