Beyond Uptime: Reliability Engineering for AI Infrastructure

Beyond Uptime: Reliability Engineering for AI Infrastructure

Sreejith Kaimal
This is a sample of the book's content.Buy on Leanpub

Table of Contents

Beyond Uptime : Reliability Engineering for AI Infrastructure›

    Disclaimer & Limitation of Liability›

    • How AI Infrastructure Actually Works
      • What is an AI Model
      • Why this requires GPU’s rather than CPU’s
      • Visualizing the AI Factory Floor
      • How Tiny Delays Compound in AI
      • The Modern AI Stack : A Glossary Through the Same Factory Lens
      • Beyond Network Rules : A Systems-Thinking Argument
    • Operation Slowburn
      • How can a system be healthy and still be failing?
    • The Synchronization Problem
      • An Intuitive Model: Barrier Synchronization
      • From the Table to the Training Run
      • Why this is unavoidable
      • A Worked Capacity-Planning Illustration
    • Operation Slowburn : A Forensic Reconstruction
      • Incident Premise and Cluster Configuration
      • The Forensic Investigation
    • Specialized Networking Technologies
      • Limits of Ordinary Networking
      • RDMA, RoCE, and InfiniBand
      • A Cautionary Case: Fast Transport Is Not Sufficient Alone
      • Comparing the Three Fabrics Along the Dimensions That Matter
      • Transport Reliability Semantics: What RDMA Guarantees and What It Does Not
    • Lossless Fabric Bottlenecks
      • The Congested Highway
      • Priority Flow Control and Cascading Pause Frames
      • Hash Polarization: A Second Congestion Mechanism
      • The Diagnostic Implication
    • The Anatomy of Silent Data Corruption
      • Transient Voltage Fluctuations & Thermal Noise
      • Cosmic Radiation & Single-Event Upsets (SEUs)
      • Manufacturing Variance & “Mercurial Cores”
      • Passive Hardware Telemetry Checklist
      • Memory Subsystem Degradation
      • The Structural Argument
    • Systematic RCA - The Failure Graph
      • The Fleet Aggregation Paradox
      • Observability as Evidence Collection
      • Systematic , Layer-by-Layer Method
      • A Divergent Presentation of the Same Failure Class
      • The Failure Graph as a Formal Structure
    • Generalizing the Incident :A Practitioner Framework
      • Why Generalization Is Possible at All
      • From Forensics to Foresight
      • Toward Automated Detection
      • A Concrete Detection Formalism: From Fleet Averages to Per-Rank Tail Statistics
      • Preventive Engineering Practices
      • Organizational Practices: Building the Team Around the Method
    • AI Investigating AI : Towards Autonomous Infrastructure
      • Machine Learning as the Investigator, Not Only the Instrument
      • Toward Autonomous AI Infrastructure
      • An example of a Harnessed Agent Cluster running an AI model
      • An Agenda for Further Research

    References›

      Beyond Uptime: Reliability Engineering for AI Infrastructure/Disclaimer & Limitation of Liability/Specialized Networking Technologies

      Specialized Networking Technologies

      https://leanpub.com/forensic-ai

      Limits of Ordinary Networking

      https://leanpub.com/forensic-ai

      RDMA, RoCE, and InfiniBand

      https://leanpub.com/forensic-ai

      A Cautionary Case: Fast Transport Is Not Sufficient Alone

      https://leanpub.com/forensic-ai

      Comparing the Three Fabrics Along the Dimensions That Matter

      https://leanpub.com/forensic-ai

      Transport Reliability Semantics: What RDMA Guarantees and What It Does Not

      https://leanpub.com/forensic-ai

      Previous

      Operation Slowburn : A Forensic Reconstruction

      Up next

      Lossless Fabric Bottlenecks

      In this chapter

      • Specialized Networking Technologies
      • Limits of Ordinary Networking
      • RDMA, RoCE, and InfiniBand
      • A Cautionary Case: Fast Transport Is Not Sufficient Alone
      • Comparing the Three Fabrics Along the Dimensions That Matter
      • Transport Reliability Semantics: What RDMA Guarantees and What It Does Not