Kick off your book project in 2 hours, get started with GhostAI in 2 hours, or do both! Free live workshops, on Zoom. You’ll leave with a real book project and a clear plan to keep going. Saturday, June 27, 2026.

Leanpub Header

Skip to main content
Go to Leanpub.com

Notes on Dynamical Systems for Actor-Critic Learning

A Dynamical Systems Approach to Reinforcement Learning Mean Dynamics

Vladyslav Prytula

An introduction to actor-critic algorithms as dynamical systems: featuring hand-computable examples, fast-slow reductions, and machine-checked Lean 4 proofs

Minimum price

Free!

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
About

About

About the Book

Notes on Attractors in Actor-Critic Learning provides a rigorous, dynamical-systems treatment of finite-state actor-critic algorithms. By modeling the joint evolution of the actor, the critic, and the policy's state distribution together on an enlarged phase space, this book moves beyond simple point-convergence to analyze the global, asymptotic behavior of learning feedback loops.

Key Features:

  • Pedagogical Foundations: Opens with a hand-computable, two-state model (Chapter 0) to ground the abstract mathematics in concrete calculations.
  • Rigorous Analysis: Establishes well-posedness, compact absorbing sets, global attractors, and fast-slow reductions.
  • Fully Formalized Proofs: Backed by an axiom-free Lean 4 formalization of the core mathematical package.
  • Real-World Applications: Applies the theoretical framework to content recommendation systems (filter-bubble attractors) and endogenous network routing.

This is an attempt to bridge reinforcement learning, control theory, and formal methods for those who want to study the deep, unified dynamics of learning.

Share this book

Categories

Computer Programming/Artificial Intelligence/PythonFunctional ProgrammingMathematicsAlgorithmFormal Methods

Feedback

Email the Author

Author

About the Author

Vladyslav Prytula

Vladyslav Prytula is an applied mathematician who has spent his career moving between dynamical systems, homogenization of PDEs and machine learning. His training began in Kharkiv under Igor Chueshov, whose attractor theory sits underneath much of this book, and continued through a PhD and postdoctoral years in partial differential equations and infinite-dimensional dynamical system: well-posedness, global attractors, homogenization , from a doctoral student in Spain to an Abel Fellow and associate professor in Norway.

These days he is Principal ML/AI research scientist / Director of ML/AI at a European e-commerce company, building search, recommendation, and agentic systems at production scale. The book came out of a stubborn conviction that the way actor-critic methods converge is best understood not as an optimization trick but as the long-time behaviour of a coupled flow — actor, critic, and state distribution moving together on one phase space — and that you can write those dynamics down precisely, prove things about them, and have a machine check the proofs.

He lives in Munich, where most of his thinking happens on long trail runs and in the mountains.

Contents

Table of Contents

Foreword

Preface

  1. What This Book Is About
  2. Who This Book Is For
  3. How The Running Example Works
  4. How To Read This Book
  5. Machine Verification

Notation And Dependencies

  1. Symbol Table
  2. Chapter Dependency Diagram
  3. Conventions

Chapter 0: The Worked Example

  1. 0.1 Why We Start With An Example
  2. 0.2 Model Overview
  3. 0.3 The Environment
  4. 0.4 The Policy
  5. 0.5 The State Distribution And The Occupancy Measure
  6. 0.6 Why The Actor Drift Does Not Close On \theta Alone
  7. 0.7 The Critic Equation
  8. 0.8 The Actor Equation
  9. 0.9 The Distribution Equation
  10. 0.10 The Full Coupled System
  11. 0.11 Forward Invariance
  12. 0.12 The Absorbing Set And Boundedness
  13. 0.13 Equilibria
  14. 0.14 The Phase Portrait
  15. 0.15 Breaking The Symmetry
  16. 0.16 What The Attractor Contains
  17. 0.17 Summary And Bridge Forward
  18. Exercises

Chapter 1: The Prerequisite Bridge

  1. 1.1 What The Example Showed And What It Left Open
  2. 1.2 From Algorithms To Flows
  3. 1.3 Semiflows
  4. 1.4 Forward Invariance
  5. 1.5 Absorbing Sets
  6. 1.6 Omega-Limit Sets
  7. 1.7 Global Attractors
  8. 1.8 Why The Enlarged State Space?
  9. 1.9 The Prescribed Closure Map
  10. 1.10 The Program Ahead
  11. 1.11 Summary Of Vocabulary
  12. Exercises

Chapter 2: The General Model

  1. 2.1 The Softmax Policy And Its Score Function
  2. 2.2 The Generator Family And The Law Equation
  3. 2.3 Occupancy Measures And The Critic Equation
  4. 2.4 The Actor Drift And Boundary Damping
  5. 2.5 The Standing Assumptions
  6. 2.6 The Full System And The Phase Space
  7. 2.7 Recovery Of The Worked Example
  8. 2.8 A Three-State Retail-To-Vet Routing Example
  9. 2.9 Summary And Bridge Forward
  10. Exercises

Chapter 3: Local Lipschitz Regularity and Well-Posedness

  1. The Regularity Principle
  2. 3.1 Local Lipschitz Continuity of the Softmax
  3. 3.2 Local Lipschitz Continuity of the Actor Drift
  4. 3.3 Local Lipschitz Continuity of the Critic Drift
  5. 3.4 Local Lipschitz Continuity of the Law Field
  6. 3.5 The Ambient Extension and Picard-Lindelöf
  7. 3.6 From Local to Global Existence
  8. 3.7 Summary and Bridge Forward
  9. Exercises

Chapter 4: A Priori Estimates

  1. 4.1 Actor-Box Forward Invariance
  2. 4.2 Simplex Forward Invariance
  3. 4.3 Critic Coercivity and the Energy Estimate
  4. 4.4 The Compact Absorbing Set
  5. 4.5 Global Existence and the Semiflow
  6. 4.6 Summary and Bridge Forward
  7. Exercises

Chapter 5: The Global Attractor

  1. 5.1 The Omega-Limit Set Revisited
  2. 5.2 Nonemptiness Of \omega(K)
  3. 5.3 Compactness Of \omega(K)
  4. 5.4 Invariance Of \omega(K)
  5. 5.5 Attraction Of Bounded Sets
  6. 5.6 Uniqueness
  7. 5.7 The Prescribed-Closure Global Attractor Theorem
  8. 5.8 What The Attractor Contains And What It Does Not Determine
  9. Exercises

Chapter 6: Bridge To The Genuine Controlled-Chain Closure

  1. 6.1 The Frozen Chain And Its Invariant Law
  2. 6.2 Uniform Exponential Mixing
  3. 6.3 Lipschitz Regularity Of The Invariant-Law Map
  4. 6.4 The Bridge Theorem
  5. 6.5 The Minorization Condition
  6. 6.6 Summary And Bridge Forward
  7. Exercises

Chapter 7: Fast-Slow Reduction

  1. 7.1 The Two-Timescale Setup
  2. 7.2 The Pathwise Tracking Estimate
  3. 7.3 Upper Semicontinuity Of Attractors
  4. 7.4 The Minorization Sufficient Condition
  5. Exercises

Chapter 8: Outlook And Open Problems

  1. 8.1 Non-Autonomous Forcing
  2. 8.2 Stochastic Perturbations
  3. 8.3 Closing Perspective
  4. Exercises

Chapter 9: From Theory to Models

  1. 9.1 What Instantiation Means
  2. 9.2 The Model Specification Protocol
  3. 9.3 Feature Design and Attractor Geometry
  4. 9.4 Generator Construction from Domain Topology
  5. 9.5 Reading the Attractor in Domain Language
  6. 9.6 Preview of the Application Chapters
  7. 9.7 Chapter Summary
  8. Exercises

Chapter 10: Recommendation Systems and Algorithmic Curation

  1. 10.1 The Recommendation Problem as a Dynamical System
  2. 10.2 Model Specification: States, Actions, and Features
  3. 10.3 Rewards and the Engagement-Diversity Tension
  4. 10.4 The Generator Family: A Y-Graph Controlled Chain
  5. 10.5 The Full Recommendation System
  6. 10.6 Equilibria and Filter Bubbles
  7. 10.7 What the Theory Reveals
  8. 10.8 Summary and Bridge Forward
  9. Exercises

Chapter 11: Network Routing Under Endogenous Traffic

  1. 11.1 From the Retail-Vet Chain to a Full Network
  2. 11.2 Model Specification: The Hub-and-Spoke Network
  3. 11.3 The Generator Family: Network Topology as Generator Structure
  4. 11.4 Reference-State Minorization and the Bridge Theorem
  5. 11.5 The Full Routing System and Its Equilibria
  6. 11.6 Attractor Structure and Routing Policy Design
  7. 11.7 Summary and Bridge to Appendix B
  8. Exercises

Appendix A: Lean Formalization Structure

  1. Verification
  2. Paper-to-Lean Mapping
  3. File Layout
  4. Key Design Decisions
  5. Scope of the Unconditional Claim
  6. Reading Order

Appendix B: Computational Methods and Phase Portrait Blueprint

  1. B.1 Numerical Integration of the Model System
  2. B.2 Equilibrium Finding and Nullcline Computation
  3. B.3 Phase Portrait Blueprint: The Chapter 0 and Section 2.8 Models
  4. B.4 Phase Portrait Blueprint: The Recommendation and Routing Models
  5. B.5 Parameter Continuation and Bifurcation Sketches

References

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub