Beyond Uptime: Reliability Engineering for AI Infrastructure

Beyond Uptime: Reliability Engineering for AI Infrastructure

Sreejith Kaimal
This is a sample of the book's content.Buy on Leanpub

Table of Contents

Beyond Uptime : Reliability Engineering for AI Infrastructure›

    Disclaimer & Limitation of Liability›

    • How AI Infrastructure Actually Works
      • What is an AI Model
      • Why this requires GPU’s rather than CPU’s
      • Visualizing the AI Factory Floor
      • How Tiny Delays Compound in AI
      • The Modern AI Stack : A Glossary Through the Same Factory Lens
      • Beyond Network Rules : A Systems-Thinking Argument
    • Operation Slowburn
      • How can a system be healthy and still be failing?
    • The Synchronization Problem
      • An Intuitive Model: Barrier Synchronization
      • From the Table to the Training Run
      • Why this is unavoidable
      • A Worked Capacity-Planning Illustration
    • Operation Slowburn : A Forensic Reconstruction
      • Incident Premise and Cluster Configuration
      • The Forensic Investigation
    • Specialized Networking Technologies
      • Limits of Ordinary Networking
      • RDMA, RoCE, and InfiniBand
      • A Cautionary Case: Fast Transport Is Not Sufficient Alone
      • Comparing the Three Fabrics Along the Dimensions That Matter
      • Transport Reliability Semantics: What RDMA Guarantees and What It Does Not
    • Lossless Fabric Bottlenecks
      • The Congested Highway
      • Priority Flow Control and Cascading Pause Frames
      • Hash Polarization: A Second Congestion Mechanism
      • The Diagnostic Implication
    • The Anatomy of Silent Data Corruption
      • Transient Voltage Fluctuations & Thermal Noise
      • Cosmic Radiation & Single-Event Upsets (SEUs)
      • Manufacturing Variance & “Mercurial Cores”
      • Passive Hardware Telemetry Checklist
      • Memory Subsystem Degradation
      • The Structural Argument
    • Systematic RCA - The Failure Graph
      • The Fleet Aggregation Paradox
      • Observability as Evidence Collection
      • Systematic , Layer-by-Layer Method
      • A Divergent Presentation of the Same Failure Class
      • The Failure Graph as a Formal Structure
    • Generalizing the Incident :A Practitioner Framework
      • Why Generalization Is Possible at All
      • From Forensics to Foresight
      • Toward Automated Detection
      • A Concrete Detection Formalism: From Fleet Averages to Per-Rank Tail Statistics
      • Preventive Engineering Practices
      • Organizational Practices: Building the Team Around the Method
    • AI Investigating AI : Towards Autonomous Infrastructure
      • Machine Learning as the Investigator, Not Only the Instrument
      • Toward Autonomous AI Infrastructure
      • An example of a Harnessed Agent Cluster running an AI model
      • An Agenda for Further Research

    References›

      Beyond Uptime: Reliability Engineering for AI Infrastructure/Overview

      Beyond Uptime: Reliability Engineering for AI Infrastructure

      Course overview

      Master AI infrastructure reliability! Debug silent GPU cluster stalls, PFC pause cascades, PCIe downtraining, and MFU drops across distributed compute.

      3 parts
      ·
      10 chapters
      Begin reading
      Download
      Part 1

      Beyond Uptime : Reliability Engineering for AI Infrastructure0 chapters

      Begin part ›
        Part 2

        Disclaimer & Limitation of Liability10 chapters

        Begin part ›
        1. How AI Infrastructure Actually Works

        2. Operation Slowburn

        3. The Synchronization Problem

        4. Operation Slowburn : A Forensic Reconstruction

        5. Specialized Networking Technologies

        6. Lossless Fabric Bottlenecks

        7. The Anatomy of Silent Data Corruption

        8. Systematic RCA - The Failure Graph

        9. Generalizing the Incident :A Practitioner Framework

        10. AI Investigating AI : Towards Autonomous Infrastructure

        Part 3

        References0 chapters

        Begin part ›