-
1. Behavioral Cloning
- 1.1 The Setting
- 1.2 The Idea
- 1.3 Formalization
- 1.4 Implementation
- 1.5 When Does BC Work?
- 1.6 Why BC Fails: The Compounding Error Problem
- 1.7 Practical Limitations for Industrial Processes
- 1.8 The BC-to-RL Bridge: DAgger
- 1.9 Summary
- 1.10 What Comes Next
-
2. The Offline RL Problem
- 2.1 From BC to RL: What Changes
- 2.2 Q-Learning: A Brief Recap
- 2.3 The Extrapolation Error Problem
- 2.4 Formal Statement: The Performance Gap
- 2.5 Distribution Shift: The Offline RL Version
- 2.6 Empirical Demonstration
- 2.7 The Core Challenge
- 2.8 Why Not Just Use BC?
- 2.9 Summary
-
3. Off-Policy Evaluation (OPE)
- 3.1 The Problem
- 3.2 Why Not Just Use the Learned Q-Function?
- 3.3 Three Families of Estimators
- 3.4 Practical Recommendations
- 3.5 Limitations
- 3.6 Summary
-
4. Conservative Q-Learning (CQL)
- 4.1 The Problem, Restated
- 4.2 The Idea
- 4.3 Formalization
- 4.4 Implementation
- 4.5 The alpha Hyperparameter
- 4.6 Intuition: The Q-Landscape
- 4.7 How to Choose α
- 4.8 CQL vs TD3+BC: A Comparison
- 4.9 Practical Tips
- 4.10 Limitations
- 4.11 Summary
-
5. Implicit Q-Learning (IQL)
- 5.1 What CQL Got Right — and What It Didn't
- 5.2 The Core Idea
- 5.3 Formalization
- 5.4 Implementation
- 5.5 What Expectile Tau Does
- 5.6 IQL vs CQL: Key Differences
- 5.7 Hyperparameter Guide
- 5.8 Practical Tips
- 5.9 Limitations
- 5.10 Summary
-
6. Policy-Constraint and Actor-Critic (TD3+BC, AWAC)
- 6.1 Where Value-Pessimism Leaves Off
- 6.2 TD3+BC: A Minimalist Policy-Regularized Approach
- 6.3 AWAC: Advantage-Weighted Actor-Critic
- 6.4 Implementation
- 6.5 Hyperparameters and Practical Tips
- 6.6 Limitations
- 6.7 Summary
-
7. Decision Transformers
- 7.1 A Different Paradigm
- 7.2 The Idea
- 7.3 Formalization
- 7.4 Implementation Sketch
- 7.5 Limitations
- 7.6 Summary
-
8. Model-Based Offline RL (MOPO, MOReL)
- 8.1 Beyond Model-Free
- 8.2 Part I: MOPO
- 8.3 Part II: MOReL
- 8.4 MOPO vs MOReL vs Model-Free
- 8.5 Hyperparameter Guide
- 8.6 Practical Tips
- 8.7 Limitations
- 8.8 Summary
-
9. Physics-Informed Offline RL
- 9.1 Why Physics?
- 9.2 Part I: Physics as Reward Shaping
- 9.3 Part II: Hybrid Dynamics Models
- 9.4 Part III: Constraint-Based Policy (Brief)
- 9.5 Part IV: When Physics Helps Most
- 9.6 Further Directions
- 9.7 Summary
-
10. Industrial Applications
- 10.1 From Benchmarks to Industry
- 10.2 The Coating Process
- 10.3 The Dataset: Operating Regimes
- 10.4 Evaluation: Industrial Metrics
- 10.5 The Physics Model
- 10.6 Algorithm 1: Behavioral Cloning
- 10.7 Algorithm 2: CQL
- 10.8 Algorithm 3: CQL + Constraint Penalty
- 10.9 Algorithm 4: HybridMOReL
- 10.10 Results
- 10.11 Engineering Decisions Revisited
- 10.12 Safe RL, Drift Detection, and Fallback
- 10.13 Summary
-
11. Explainability in Offline RL
- 11.1 Why Explainability Matters in Industrial Offline RL
- 11.2 SHAP Background
- 11.3 Three Levels of Explanation
- 11.4 The Explainer Class
- 11.5 Visualization
- 11.6 Consistency Check
- 11.7 SHAP in Production: Practical Considerations
- 11.8 What SHAP Cannot Tell You
- 11.9 Beyond SHAP: Causal AI in Offline RL
- 11.10 Summary
- 11.11 Appendix 11.A: Choosing Between SHAP Variants
-
12. Offline RL for Tool-Using LLM Agents
- 12.1 Why This Chapter Exists
- 12.2 Agent Traces as Offline RL Data
- 12.3 MDP / POMDP Formulation
- 12.4 The Policy Interface Determines the Training Method
- 12.5 BC / SFT as the Baseline
- 12.6 Offline RL Beyond SFT
- 12.7 Credit Assignment in Reasoning and Tool Use
- 12.8 OPE for Agents
- 12.9 LLMs as Annotators, Generators, and Critics
- 12.10 Failure Modes
- 12.11 Toy Example: Offline RL for a Calculator Agent
- 12.12 From Agentic Offline RL to Offline MARL
- 12.13 Practical Logging Schema
- 12.14 Limitations and Honest Scope
-
13. Conclusion and Future Directions
- 13.1 What This Book Built
- 13.2 When Not to Use Offline RL
- 13.3 About the Code in This Book
- 13.4 What Offline RL Cannot Yet Guarantee
- 13.5 Emerging Directions
- 13.6 Industrial Deployment: Gated Readiness
- 13.7 Closing Remarks
-
A Algorithm Selection Guide
- A.1 Quick selection table
- A.2 Source anchors (key methods)
- A.3 Code in this repository
- A.4 Glossary
- A.5 Suggested reading order for practitioners
Offline RL
From Theory to Industrial Practice
What to do when you have years of logs and no permission to experiment: offline RL from theory to industrial practice.
With Membership
Free!
$20.00
You pay
Author earns
PDF
EPUB
About
About the Book
Offline reinforcement learning trains a control policy from data that has already been logged, with no experiments on the real system. It matters wherever trial and error is expensive or unsafe, as in an industrial plant.
The book explains why the problem is hard and how it is solved. It starts with simply imitating the operator and with the central trap: the learned values overrate actions the data never shows. It then covers evaluating a policy from logs and the main families of methods: conservative value estimates, policy constraints, Decision Transformers and learning with a model of the environment. The last chapters turn to industrial practice: building process physics into the model, a larger case study checked over many runs, explaining decisions, and LLM agents.
Each chapter gives the idea, the mathematics, working code and an honest look at where the method breaks down. The code is for learning: it shows how the algorithms work and is not meant as a production library.
Author
About the Author
Artem Zvyagintsev
Artem Zvyagintsev is a data scientist and independent researcher working on offline and multi-agent reinforcement learning.
Over twenty years, he has worked as a senior software engineer and data engineer, and most recently led data science in manufacturing.
His current research includes model-based offline RL, post-training of open-weight language models, and the behavior of LLM agents in multi-agent settings. He also writes about differentiable programming and category theory.
Translations
Translations
Languages
Contents
Table of Contents
Get the free sample chapters
Click the buttons to get the free sample in PDF or EPUB, or read the sample online here
The Leanpub 60 Day 100% Happiness Guarantee
Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms...
Earn $8 on a $10 Purchase, and $16 on a $20 Purchase
We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.
(Yes, some authors have already earned much more than that on Leanpub.)
Learn more about writing on Leanpub
Free Updates. DRM Free.
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them
Write and Publish on Leanpub
You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!
Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.
Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.