Preface

Consider a platform that recommends content to users. It promotes items by a learned strategy, keeps a running estimate of how valuable each recommendation turns out to be, and–whether or not it tracks this explicitly–steadily reshapes which users keep coming back. A typical actor-critic algorithm formalizes these three coupled quantities: the policy weights, a value-function estimate, and the distribution of states the agent visits. These notes ask a simple question: if we run the update long enough, where do all three quantities end up?

The answer we develop is a set of theorems for the deterministic mean dynamics of finite-state actor-critic systems. The system is studied on an enlarged state space that includes actor parameters, critic variables, and policy-induced distributions. The central asymptotic object is a compact global attractor for this enlarged-state flow. The notes build the attractor theorem from scratch, connect it to the underlying controlled Markov chain, and show that a fast-slow reduction justifies replacing the dynamic state distribution by its stationary limit when the chain mixes quickly.

The notes are written for a graduate student, or a strong advanced undergraduate, who has:

• a first course in reinforcement learning at roughly the Sutton-Barto level: MDPs, Bellman equations, policy gradient, actor-critic
• a first course in ODEs or dynamical systems, possibly with introductory semigroup language and basic attractor terminology
• solid linear algebra and real analysis

The reader is assumed to have met the terms above in a course, but not to know how the two languages fit together. The notes build the bridge from both directions: for RL readers, they explain why a global attractor says more than “the algorithm converges”; for dynamical-systems readers, they explain why actor, critic, and distribution variables are genuine state coordinates rather than auxiliary variables.

Chapter 0 introduces a hand-computable model with two states, two actions, a scalar actor parameter, a scalar critic parameter, and an explicit simplex coordinate for the distribution variable. This two-state system is the example the book keeps returning to: every chapter uses it to verify that the general theory produces numbers the reader can check by hand.

Starting in Chapter 2, a second running example appears: a three-state policy-routing model that already looks like a real application. The two examples play complementary roles: the two-state model verifies every step of the core deterministic theory by hand, while the routing model returns in Chapters 6 and 7 to illustrate the bridge theorem and the fast-slow reduction, reappears in Chapter 9 as a protocol case study, and grows into the five-state network of Chapter 11.

The chapters are designed to be read in order:

• Chapter 0 introduces the worked example and the three-quantity coupling.
• Chapter 1 bridges RL and dynamical-systems language.
• Chapters 2—5 develop the core deterministic theory: the general model, well-posedness, a priori estimates, and the global attractor.
• Chapter 6 supplies the bridge theorem connecting the prescribed closure map to the invariant law of the controlled Markov chain.
• Chapter 7 proves the fast-slow reduction: finite-time tracking, upper semicontinuity of attractors, and the minorization sufficient condition–a checkable hypothesis on the chain that guarantees the fast mixing the reduction needs.
• Chapter 8 discusses outlook: non-autonomous forcing, stochastic perturbations, and open problems.
• Chapter 9 develops the Model Specification Protocol: a systematic procedure for translating a domain problem into the Chapter 2 template.
• Chapter 10 instantiates the protocol on a content recommendation platform: filter bubbles appear as boundary equilibria, and reward design controls the regime in which bubble lock-in becomes attracting.
• Chapter 11 extends the routing chain from Chapter 2, Section 2.8 to a five-state hub-and-spoke network whose equilibrium set contains a balanced routing configuration and two spoke lock-in configurations, and shows how link-reward design decides which regime the network settles into.

Chapters 0—8 form the theoretical core and stand on their own, independently of the application chapters. Chapters 9—11 apply the proved theory to concrete domains and are best read in order after Chapter 8: Chapter 9 defines the Model Specification Protocol that Chapters 10 and 11 execute, and Chapter 11 builds on Chapter 10’s analysis.

Appendix A describes the Lean 4 formalization for readers who wish to inspect the machine-checked proofs. Appendix B provides computational methods and a phase-portrait blueprint for the models developed in the main text.

The main theorems developed in Chapters 2—7–including the global attractor theorem, the frozen invariant-law proposition, the finite-time tracking theorem, and the upper-semicontinuity corollary–have been formalized in the Lean 4 proof assistant with no axioms, placeholders, or proof stubs. The main mathematical claims are machine-checked, while expository prose, numerical illustrations, and exercises remain outside that verified boundary. The Lean development was written with assistance from OpenAI Codex and Anthropic Claude. These tools helped with proof engineering and refactoring, but the verification claim rests on Lean’s kernel checking the final proof scripts, not on language-model output. The Lean formalization structure is described in Appendix A.