Leanpub Header

Skip to main content
Go to Leanpub.com

Crash-Proof AI Agents

Durable Execution for LLM Agents, Tested in the Lab

Crash-Proof AI Agents
Ground Truth Books

Kill an AI agent mid-refund and the retry bills every model call again and refunds twice. This book crashes one agent on purpose and recovers it on DBOS, Temporal, Restate and Prefect, with the real output of every run.

Minimum price

$19.00

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
About

About

About the Book

Your support agent looks up an order, refunds EUR 49, and is about to email the customer when the container is killed. The retry starts from nothing. It pays for every model call again and refunds the customer a second time.

Durable execution engines promise to fix this, but each one journals only what you tell it to. This book takes one reference agent, crashes it at known points, and recovers it on DBOS, Temporal, Restate and Prefect. Every output shown is a captured run, so you can see which calls were saved, which ran again, and where the refund still doubled.

  • Why DBOS refunds twice until the tools are steps, and why Temporal does not
  • Why the model call in flight at a crash is billed twice on every engine
  • Idempotency keys that let the provider drop the second refund
  • Retries and 429s: which layer honors Retry-After, and which ignores it
  • Human approvals that survive a crash while the approver is still deciding
  • Versioning a workflow that is paused mid-run
  • Traces that tie a crashed attempt to its recover

It ends with a pytest suite that kills the agent at five points and checks the refund, the email and the billed calls, and with a step-by-step retrofit of a plain agent, checked by that suite after each change. The full lab code is printed in an appendix.

Researched and drafted with AI assistance by Ground Truth Books. Every output shown was produced by real runs in a pinned container, the model in those runs is a scripted stand-in, and the text was checked against the captures and the pinned library sources. Product names are used descriptively; this book is not affiliated with or endorsed by Temporal, DBOS, Restate, Prefect, Pydantic or any other vendor named.

Share this book

Categories

Artificial IntelligencePythonSoftware ArchitectureResiliencySoftware Engineering

Feedback

Contact the author

Author

About the Author

Ground Truth Books

Ground Truth Books publishes practical technical books on computer science, IT tools, data science, machine learning, software engineering and AI.

The name comes from machine learning, where "ground truth" means the real, verified answers you check a model against. That's how these books are made. They're researched and drafted with AI assistance, and then checked against reality. Examples are run against real captures and data in a lab, and those runs are cited like any other source, so you can see which output came straight from the tool. Every claim is cited to its source, with official documentation, specifications and source code preferred over blog posts.

Each book is built around doing the work. Chapters end with exercises, and an appendix gives worked answers you can reproduce on your own machine, using the same freely available data and captures.

Books are updated when the tools change. If you find an error, please report it: a corrected edition is free for every reader, which is one of the best things about Leanpub.

Contents

Table of Contents

About This Book

Introduction

  1. Why agents crash
  2. The reference agent
  3. What you will learn
  4. How to use this book
  5. The Problem

The reference agent and its baseline

  1. Why agents die in the wild
  2. Timeouts and stalled calls
  3. Rate limits and 429s
  4. Deploys and OOM kills
  5. The failure modes in-lab
  6. What a restart loses
  7. The ticket agent flow
  8. The stand-in model and ledger
  9. Running the baseline lab
  10. kill -9 after the refund
  11. Reading ledger.log

Durable execution fundamentals

  1. Workflows, steps, and activities
  2. Replay and determinism
  3. Exactly-once versus at-least-once
  4. What the engine journals
  5. Why replay must not re-bill
  6. Durable Execution with DBOS

DBOS Transact on Postgres

  1. Postgres as the engine
  2. Workflows and workflow IDs
  3. The DBOSDurability capability
  4. The 3.0 schema break
  5. Running the DBOS lab

DBOS step tools for tool calls

  1. Model calls versus step calls
  2. Wrapping tools with DBOS.step
  3. dbos-crash and dbos-step-tools compared
  4. Reading operation_outputs
  5. Why the refund runs once

Recovering interrupted DBOS runs

  1. What a crash leaves behind
  2. How workflow recovery works
  3. Recovering pending workflows
  4. Workflow IDs and idempotency
  5. Upgrading code during recovery
  6. Operating recovery in production
  7. Durable Execution with Temporal

Temporal workers and task queues

  1. The Temporal service and workers
  2. Workflows versus activities
  3. Polling task queues
  4. Event history persistence
  5. Starting a Python worker

Temporal tool calls as activities

  1. The TemporalDurability capability
  2. Tool calls as activities
  3. The temporal-crash lab
  4. The in-flight request retried
  5. Reading the event history

Replay safety in Temporal

  1. Determinism on replay
  2. Non-determinism failures
  3. Safe payload conversion
  4. Replay-safe tracing
  5. Replay testing your workflows
  6. Durability Patterns

Idempotent tools and idempotency keys

  1. Where double effects come from
  2. Idempotency keys on tools
  3. kill -9 during the refund
  4. Treating calls as retries
  5. The ledger after recovery

Retries, fallbacks, and replay billing

  1. Configuring retries per step
  2. Fallback models and providers
  3. Honoring Retry-After
  4. When replay must not re-bill
  5. Coupling retries to durability

Human-in-the-loop waits

  1. Why waits break plain agents
  2. Durable sleep and signals
  3. Signals in Temporal
  4. Human approval across a restart
  5. Approval across a kill -9

Versioning workflows in flight

  1. When code changes mid-run
  2. workflow.patched and deprecate_patch
  3. Worker and task-queue versioning
  4. Upgrading DBOS workflows
  5. Versioning the reference agent

Observability across restarts

  1. Tracing what an agent did
  2. Spans and events across recoveries
  3. Replay-safe exporters
  4. Correlating attempts by run id
  5. Tracing the crash lab
  6. Putting It to Work

Crash tests as a CI suite

  1. Kill points as pytest tests
  2. Recovering with a fresh process
  3. Asserting the refund occurs once
  4. Ledger counts after recovery
  5. DBOS step-name checks
  6. Temporal replay tests
  7. Regressions the suite catches
  8. A CI job

Retrofitting an existing agent, end to end

  1. The retrofit roadmap
  2. DBOSDurability capability first
  3. Tools as DBOS steps
  4. Adding the idempotency key
  5. Adding tracing
  6. The Temporal upgrade path
  7. From double to single refund
  8. Replaying both histories
  9. Alternatives and Decisions

The Restate option

  1. Restate’s virtual objects
  2. Durable steps in Python
  3. Restate with pydantic-ai
  4. Recovery and journaling model
  5. Running the Restate lab

The Prefect option

  1. Flows, tasks, and results
  2. Result persistence and retries
  3. Resuming a crashed flow
  4. The Prefect durability capability
  5. Running the Prefect lab

Choosing your durability stack

  1. What each engine journals
  2. When files-plus-resume is enough
  3. Comparing Restate, Inngest, Hatchet
  4. Cost of complexity versus guarantees
  5. Decision guide for your agent

Conclusion

  1. What the lab proved
  2. What you can now build
  3. Where to go next

Appendix A: LangGraph checkpointers

  1. What a checkpointer is
  2. Backends
  3. The saver interface
  4. Durability modes
  5. Recovery by thread_id
  6. Interrupts and approval
  7. What a checkpointer does not do
  8. Between LangGraph and the engines

Appendix B: OpenAI Agents SDK sessions

  1. What a session is
  2. The Session protocol
  3. Built-in backends
  4. Retrieval and compaction
  5. The durability boundary
  6. Sessions and human approval
  7. What to reach for instead

Appendix C: The lab code

  1. The container
  2. ticket.py
  3. dbos_app.py
  4. temporal_app.py
  5. history.py
  6. test_crashes.py
  7. test_replay.py
  8. provider_api.py
  9. tracing.py
  10. prefect_app.py
  11. restate_app.py

Glossary

References

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

Download Sample PDFDownload Sample EPUB

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub