Foreword

These notes grew out of a mismatch that appears quickly in actor-critic models. The long-time question is often phrased as if the policy parameter were the only variable that matters. But the actor is not alone. The critic moves with it, and the distribution of states seen by the algorithm is shaped by the policy currently being used. After averaging out the sampling noise, those three quantities form a coupled deterministic system.

The point of the book is to take that coupled system literally. We first ask what the phase space is, what flow it carries, which estimates keep the flow bounded, and which invariant sets organize its long-time behavior. A compact global attractor is the object that answers that question. It records all bounded asymptotic behavior of the enlarged system, not only the cases where the actor happens to converge to a single point.

The development stays finite-state on purpose. Chapter 0 keeps a two-state model in view so that the main quantities can be computed by hand. The later chapters move from that example to the general finite-state actor-critic-law system, then to the controlled-chain closure and the fast-slow reduction. The application chapters return to recommendation and routing models, where the abstract variables have concrete interpretations.

There are two companion resources. The research article, “Global attractors and fast-slow reduction for finite-state actor-critic mean dynamics,” gives the compressed theorem package and is available on arXiv at https://arxiv.org/abs/2604.13259. The public repository at https://codeberg.org/VladP/Notes-on-Dynamical-Systems-for-Actor-Critic-Learning contains the manuscript source, Lean formalization, and reproducibility scripts. Appendix A explains how the Lean files correspond to the theorem statements. Appendix B describes the numerical phase-portrait computations.

This is a bridge text. It assumes the reader has seen reinforcement learning and ODE or dynamical-systems language separately, but not that the interface between them is already familiar. The aim is to make the enlarged-state viewpoint precise enough to prove, concrete enough to compute with, and visible enough to reuse.