Notes on Dynamical Systems for Actor-Critic Learning

Notes on Dynamical Systems for Actor-Critic Learning

Vladyslav Prytula
Buy on Leanpub

Table of Contents

Notes on Dynamical Systems for Actor-Critic Learning

  • Foreword
  • Preface
    • What This Book Is About
    • Who This Book Is For
    • How The Running Example Works
    • How To Read This Book
    • Machine Verification
  • Notation And Dependencies
    • Symbol Table
    • Chapter Dependency Diagram
    • Conventions
  • Chapter 0: The Worked Example
    • 0.1 Why We Start With An Example
    • 0.2 Model Overview
    • 0.3 The Environment
    • 0.4 The Policy
    • 0.5 The State Distribution And The Occupancy Measure
    • 0.6 Why The Actor Drift Does Not Close On Alone
    • 0.7 The Critic Equation
    • 0.8 The Actor Equation
    • 0.9 The Distribution Equation
    • 0.10 The Full Coupled System
    • 0.11 Forward Invariance
    • 0.12 The Absorbing Set And Boundedness
    • 0.13 Equilibria
    • 0.14 The Phase Portrait
    • 0.15 Breaking The Symmetry
    • 0.16 What The Attractor Contains
    • 0.17 Summary And Bridge Forward
    • Exercises
  • Chapter 1: The Prerequisite Bridge
    • 1.1 What The Example Showed And What It Left Open
    • 1.2 From Algorithms To Flows
    • 1.3 Semiflows
    • 1.4 Forward Invariance
    • 1.5 Absorbing Sets
    • 1.6 Omega-Limit Sets
    • 1.7 Global Attractors
    • 1.8 Why The Enlarged State Space?
    • 1.9 The Prescribed Closure Map
    • 1.10 The Program Ahead
    • 1.11 Summary Of Vocabulary
    • Exercises
  • Chapter 2: The General Model
    • 2.1 The Softmax Policy And Its Score Function
    • 2.2 The Generator Family And The Law Equation
    • 2.3 Occupancy Measures And The Critic Equation
    • 2.4 The Actor Drift And Boundary Damping
    • 2.5 The Standing Assumptions
    • 2.6 The Full System And The Phase Space
    • 2.7 Recovery Of The Worked Example
    • 2.8 A Three-State Retail-To-Vet Routing Example
    • 2.9 Summary And Bridge Forward
    • Exercises
  • Chapter 3: Local Lipschitz Regularity and Well-Posedness
    • The Regularity Principle
    • 3.1 Local Lipschitz Continuity of the Softmax
    • 3.2 Local Lipschitz Continuity of the Actor Drift
    • 3.3 Local Lipschitz Continuity of the Critic Drift
    • 3.4 Local Lipschitz Continuity of the Law Field
    • 3.5 The Ambient Extension and Picard-Lindelöf
    • 3.6 From Local to Global Existence
    • 3.7 Summary and Bridge Forward
    • Exercises
  • Chapter 4: A Priori Estimates
    • 4.1 Actor-Box Forward Invariance
    • 4.2 Simplex Forward Invariance
    • 4.3 Critic Coercivity and the Energy Estimate
    • 4.4 The Compact Absorbing Set
    • 4.5 Global Existence and the Semiflow
    • 4.6 Summary and Bridge Forward
    • Exercises
  • Chapter 5: The Global Attractor
    • 5.1 The Omega-Limit Set Revisited
    • 5.2 Nonemptiness Of
    • 5.3 Compactness Of
    • 5.4 Invariance Of
    • 5.5 Attraction Of Bounded Sets
    • 5.6 Uniqueness
    • 5.7 The Prescribed-Closure Global Attractor Theorem
    • 5.8 What The Attractor Contains And What It Does Not Determine
    • Exercises
  • Chapter 6: Bridge To The Genuine Controlled-Chain Closure
    • 6.1 The Frozen Chain And Its Invariant Law
    • 6.2 Uniform Exponential Mixing
    • 6.3 Lipschitz Regularity Of The Invariant-Law Map
    • 6.4 The Bridge Theorem
    • 6.5 The Minorization Condition
    • 6.6 Summary And Bridge Forward
    • Exercises
  • Chapter 7: Fast-Slow Reduction
    • 7.1 The Two-Timescale Setup
    • 7.2 The Pathwise Tracking Estimate
    • 7.3 Upper Semicontinuity Of Attractors
    • 7.4 The Minorization Sufficient Condition
    • Exercises
  • Chapter 8: Outlook And Open Problems
    • 8.1 Non-Autonomous Forcing
    • 8.2 Stochastic Perturbations
    • 8.3 Closing Perspective
    • Exercises
  • Chapter 9: From Theory to Models
    • 9.1 What Instantiation Means
    • 9.2 The Model Specification Protocol
    • 9.3 Feature Design and Attractor Geometry
    • 9.4 Generator Construction from Domain Topology
    • 9.5 Reading the Attractor in Domain Language
    • 9.6 Preview of the Application Chapters
    • 9.7 Chapter Summary
    • Exercises
  • Chapter 10: Recommendation Systems and Algorithmic Curation
    • 10.1 The Recommendation Problem as a Dynamical System
    • 10.2 Model Specification: States, Actions, and Features
    • 10.3 Rewards and the Engagement-Diversity Tension
    • 10.4 The Generator Family: A Y-Graph Controlled Chain
    • 10.5 The Full Recommendation System
    • 10.6 Equilibria and Filter Bubbles
    • 10.7 What the Theory Reveals
    • 10.8 Summary and Bridge Forward
    • Exercises
  • Chapter 11: Network Routing Under Endogenous Traffic
    • 11.1 From the Retail-Vet Chain to a Full Network
    • 11.2 Model Specification: The Hub-and-Spoke Network
    • 11.3 The Generator Family: Network Topology as Generator Structure
    • 11.4 Reference-State Minorization and the Bridge Theorem
    • 11.5 The Full Routing System and Its Equilibria
    • 11.6 Attractor Structure and Routing Policy Design
    • 11.7 Summary and Bridge to Appendix B
    • Exercises
  • Appendix A: Lean Formalization Structure
    • Verification
    • Paper-to-Lean Mapping
    • File Layout
    • Key Design Decisions
    • Scope of the Unconditional Claim
    • Reading Order
  • Appendix B: Computational Methods and Phase Portrait Blueprint
    • B.1 Numerical Integration of the Model System
    • B.2 Equilibrium Finding and Nullcline Computation
    • B.3 Phase Portrait Blueprint: The Chapter 0 and Section 2.8 Models
    • B.4 Phase Portrait Blueprint: The Recommendation and Routing Models
    • B.5 Parameter Continuation and Bifurcation Sketches
  • References
Notes on Dynamical Systems for Actor-Critic Learning/overview

Notes on Dynamical Systems for Actor-Critic Learning

course_overview

A dynamical-systems treatment of actor-critic learning, covering global attractors, controlled-chain closure, fast-slow reduction, and Lean verification

count_chapters
begin_reading
download
p_implied_book_part_name

Notes on Dynamical Systems for Actor-Critic Learning18 chapters

Begin ›
  1. Foreword

  2. Preface

  3. Notation And Dependencies

  4. Chapter 0: The Worked Example

  5. Chapter 1: The Prerequisite Bridge

  6. Chapter 2: The General Model

  7. Chapter 3: Local Lipschitz Regularity and Well-Posedness

  8. Chapter 4: A Priori Estimates

  9. Chapter 5: The Global Attractor

  10. Chapter 6: Bridge To The Genuine Controlled-Chain Closure

  11. Chapter 7: Fast-Slow Reduction

  12. Chapter 8: Outlook And Open Problems

  13. Chapter 9: From Theory to Models

  14. Chapter 10: Recommendation Systems and Algorithmic Curation

  15. Chapter 11: Network Routing Under Endogenous Traffic

  16. Appendix A: Lean Formalization Structure

  17. Appendix B: Computational Methods and Phase Portrait Blueprint

  18. References