Chapter 0: The Worked Example

Consider a platform that recommends content to users. At any moment, the platform maintains three things: a strategy for which items to promote, a running estimate of how valuable each recommendation is (based on accumulated engagement data), and a picture of the user population — who is visiting, what they want, how their browsing patterns are distributed across different user types.

These three quantities are coupled in a way that creates a feedback loop. When the platform changes its strategy, users see different content, which changes their behavior, which shifts the population mix. The new population mix makes some recommendations more or less effective, which updates the value estimates. And updated value estimates feed back into the next strategy revision. The loop closes on itself: strategy influences population, population influences value estimates, value estimates influence strategy.

This feedback pattern is not special to recommendation systems. It appears whenever a learning agent’s behavior influences the environment it learns from: a trading algorithm that moves prices by its own volume, a routing policy that redirects traffic into new patterns, a treatment protocol that shifts the distribution of patient health states. In each case, the distribution of situations the agent encounters is not a fixed background — it responds to the agent’s choices, and the agent’s choices respond to it.

In reinforcement learning, these three quantities have standard names. The strategy is a policy controlled by tunable parameters — the actor. The running value estimate is the critic. And the distribution of states the agent visits is the state distribution. A standard actor-critic algorithm updates all three at once, and after averaging out the stochastic noise inherent in individual interactions, the updates become a deterministic system of coupled differential equations.

Throughout these notes, that deterministic mean-field ODE is the primary object. We are not analyzing the raw stochastic recursion step by step. We are studying the averaged flow that the stochastic algorithm tracks on finite time horizons and whose attractors govern its asymptotic possibilities.

These notes ask a simple question: if we run the system long enough, where do all three quantities end up?

Answering that question rigorously will eventually require softmax policies, occupancy measures, matrix-valued critic coefficients, and an explicit confinement mechanism for the actor. Before those formal ingredients appear, we want to see the smallest nontrivial finite-dimensional instance of that ODE, with one actor coordinate, one critic coordinate, and a two-state law, so that every formula can be checked by hand.

The purpose of this chapter is therefore to fix the simplest nontrivial instance of the model and to compute everything explicitly. Later chapters will return to it repeatedly: to verify abstract lemmas, to check that the general definitions specialize correctly, and to see qualitative phenomena that the formalism alone might obscure.

By the end of the chapter, we will have seen what the state space looks like, why the actor drift does not generally close on the actor parameter alone, where the dissipative mechanism lives, and what kinds of long-time behavior the system makes visible.

The full actor-critic systems that motivate these notes are usually much larger than the model of this chapter. In a neural-network implementation, the actor parameter may have thousands or millions of coordinates, the critic parameter may be another large vector, and the state law may live on a high-dimensional simplex. Chapter 0 deliberately compresses that larger picture to its smallest nontrivial mean-field ODE:

• two states and two actions,
• one actor coordinate,
• one critic coordinate,
• one scalar law coordinate , because in a two-state simplex the second component is .

The point of the reduction is to make every coupling visible by hand. Chapter 2 restores the general finite-state model with -dimensional actor, -dimensional critic, and -state distribution, and Section 2.8 works through a three-state routing example where the chain dynamics are explicit.

Two modeling moves are on the table from the start.

First, we replace the stochastic actor-critic recursion by its deterministic mean-field ODE. Second, within that ODE class, we choose a deliberately low-dimensional worked example. The first move explains why differential equations appear at all. The second explains why Chapter 0 speaks in scalar coordinates.

With that framing in place, the worked example has three evolving coordinates and five groups of fixed data. Before defining anything precisely, here is the full inventory.

State coordinates of the worked example (what evolves in time):

• Actor parameter : lives in , is confined to , and controls the policy.
• Critic parameter : lives in and estimates the value function.
• State distribution : lives in and records the fraction of time in state 1.

Data (what we choose once and hold fixed):

• Finite environment: the states and actions .
• Rewards: a number for each state-action pair.
• Actor features: a number that links to the policy.
• Critic features: a number that links to value estimates.
• Closure map: a rule that assigns a reference distribution to each .

A reader coming from the dynamical-systems side may still ask why an ODE appears at all. The underlying actor-critic algorithm is a stochastic discrete-time recursion, and we replace it by its deterministic mean-field limit — the averaged drift obtained after smoothing out the interaction noise. Chapter 1 returns to that stochastic-approximation bridge and explains what the ODE does and does not tell us about the original algorithm. In Chapter 0 we take the resulting mean-field ODE as the object to compute explicitly.

In the general model, the ODE lives on . In the present reduction, the environment has only two states and two actions, the actor and critic each have one coordinate, and the simplex is one-dimensional. So the worked example lives in . The distribution variable is not a derived equilibrium statistic frozen in advance; it is a genuine state coordinate.

The rest of the chapter defines each ingredient, writes out the ODE, and studies its structural properties. In the symmetric example studied first, an accidental cancellation leaves the subsystem autonomous while relaxes toward a -dependent target. Section 0.15 breaks that symmetry and restores the full three-way coupling.

We work with the smallest nontrivial finite-state controlled environment:

• States: .
• Actions: .

At each state , the agent chooses an action . The choice is governed by a policy that depends on the actor parameter . The environment responds with a reward that depends on the state-action pair.

We fix the following rewards:

• At state , the rewards are and .
• At state , the rewards are and .

The reward structure is simple: each state has a “matching” action that yields reward , and the other action yields reward . In the recommendation-system language of Section 0.1, “matching” means the platform promotes the right content for the current audience: type-1 users respond to content type 1, and type-2 users respond to content type 2. A match generates engagement (reward 1); a mismatch generates none (reward 0). The agent’s task is to learn which action fits which state, using only the cumulative reward signal.

This symmetric choice will produce clean formulas. Later in the chapter, we will break the symmetry to see how the coupling changes.

The policy maps the actor parameter to a probability distribution over actions at each state. We use a softmax (Boltzmann) policy with temperature and linear logits.

The softmax is the standard differentiable bridge between a real-valued parameter and a probability distribution over finitely many actions. For our two-action setup with antisymmetric features, the entire policy will be controlled by a single scalar : positive pushes the agent toward the matching action at each state (specialization), negative pushes it toward the wrong action (anti-specialization), and produces a uniform policy that ignores state identity. The scalar is a single knob controlling how decisively the platform commits to tailoring its recommendations.

Actor features. Fix the following feature vectors (scalars, since ):

• At state , the actor features are and .
• At state , the actor features are and .

These features have an antisymmetric structure: the feature at state for action is the negative of the feature at state for the same action. The sign convention means that increasing pushes the policy toward action at state and toward action at state .

Logits. For each state-action pair, the logit is

Explicitly:

Softmax policy. With temperature , the policy at state is

Because there are only two actions and the logits differ by sign, the softmax reduces to the standard logistic sigmoid. Define

Then:

The function is the probability of the “matching” action at either state: action at state , or action at state . When , one has , so the policy favors the matching action. When , the policy is uniform: . Since is a sigmoid, it is strictly increasing, smooth, and satisfies for every finite .

In the recommendation system, when is large and positive, the platform recommends content type 1 to type-1 users and content type 2 to type-2 users with high confidence — it is a specialist. When , the platform hedges, recommending both content types with equal probability regardless of the audience. The function therefore measures the quality of the platform’s targeting: near 1 means precise recommendations, near 1/2 means the platform is guessing.

We will often write for the complementary probability.

The policy tells us how the agent acts at each state, but it does not tell us how often the agent visits each state. That information is carried by the state distribution , a probability vector on .

Because , the distribution is determined by a single scalar:

In the recommendation system, is the share of type-1 users in the current audience. It is not a fixed background statistic — it changes as the platform’s strategy changes, because different recommendation policies attract different user mixes.

The state distribution simplex is therefore the interval in the coordinate .

The occupancy measure. Many quantities in reinforcement learning — rewards, gradients, critic errors — are expectations over state-action pairs. To compute any such expectation, we need to know two things: how often the agent is in each state, and what action it takes there. The occupancy measure packages both into a single probability distribution on .

Concretely, if the agent is in state with probability and then picks action with probability , the probability of seeing the pair is just the product:

This is the occupancy measure . It depends on two inputs: the policy parameter (which determines how the agent acts) and the state distribution (which determines where the agent is). Any expected reward, any gradient, any critic target can be written as a sum weighted by .

In our example:

where and . These four entries are nonnegative and sum to , so is a probability measure on .

There is an important point to absorb early. The occupancy is determined by the current policy parameter and the current state distribution . At this stage, nothing requires to be the invariant law of any Markov chain. The variable is a genuine dynamical state coordinate that evolves in time, and is its derived product with the policy. This distinction will matter throughout the notes.

Before writing down the full system, let us see informally why the mean actor update cannot be expressed as a function of alone. In a policy-gradient-type update, the actor drift has the schematic form

where the advantage-like quantity is a signal combining the reward, the critic estimate, and an entropy penalty — defined precisely in Section 0.8. Even in our scalar example, the occupancy depends on , and the advantage-like quantity may depend on a critic variable . If we try to eliminate and by assuming they are always at their equilibrium values, we are making a substitution that has not been justified. In general, and evolve on their own timescales and may not have reached equilibrium. The correct description is therefore a coupled system in the three variables .

In the specific symmetric example we are about to write down, an accident of the feature structure will cause the -dependence in the actor drift to cancel. We will exploit that simplification for the initial analysis, and then immediately break the symmetry to restore the full coupling. The point of the general theory, however, stands: the actor drift is not closed on alone whenever the occupancy or the critic introduces state dependence, and treating the system as if it were scalar in can miss genuine dynamical phenomena.

The critic variable (a scalar, since ) is an auxiliary variable that tracks how much reward the current policy is earning. In the recommendation system, it is the platform’s running scorecard: a summary of how well the current recommendation strategy is performing, updated continuously as engagement data arrives. The critic feeds into the actor drift — it is the signal that tells the actor whether to specialize further or pull back.

In these notes, the critic equation is introduced directly as a linear response law rather than derived from a Bellman or temporal-difference recursion. We are concerned only with the deterministic mean-field system and its structural properties; the stochastic-approximation origin of the critic update is part of the background the reader brings.

Critic features. Fix:

• At state , the critic features are and .
• At state , the critic features are and .

These features assign weight to the matching state-action pair and weight to the other. With the regularization constant , the critic coefficients are:

The sum has four terms, but the and terms vanish: and . Only the matching-action terms survive:

Both and depend on through , but the dependence on has cancelled because the matching-action occupancy weights are at state and at state , and these sum to . This is a direct consequence of the feature symmetry. In the general model, or when the symmetry is broken, both and will depend on .

The critic equation is

For frozen , this is a scalar linear ODE with coefficient . Because for every , the equation is uniformly coercive: the critic relaxes exponentially toward its unique equilibrium

Let us record a few values:

• If , then and .
• If , then and .
• If , then and .
• If , then and .

These values show the critic tracking the policy’s effectiveness. When is small and the policy is nearly uniform, the agent sees the matching action succeed only about half the time, so the critic’s estimate is low. As grows and the policy concentrates on matching, the reward rate rises and climbs toward . It never reaches because the regularization constant pulls the critic toward zero, creating a permanent gap. The critic is therefore a running scorecard: it reflects how much reward the current policy is actually generating, discounted by the regularization.

The equilibrium value is strictly between and for every finite , and it increases monotonically with .

The actor parameter lives in the box

The raw actor drift is a score-type field that sums over all state-action pairs the product of the occupancy, an advantage-like reward signal, and the policy score.

Score functions. The score is the direction in parameter space that makes action more likely at state . It is the building block of any policy-gradient update: the agent adjusts along the score, weighted by how good the action turned out to be. In the softmax policy with temperature , the score at state-action pair is

At state :

Therefore

At state :

Therefore

The same antisymmetric feature structure makes this state- computation a mirror image of the state- case, and we will see later in this section, when the drift simplifies, that this symmetry is what causes the -dependence to cancel.

Entropy-regularized advantage. For each state-action pair, the advantage-like signal appearing in the actor drift is

The first two terms measure how good action is at state : the immediate reward plus the critic’s estimate. The third term, , is the entropy penalty. It is large and positive when the policy assigns low probability to action (because is a large negative number), and small when the policy already favors . The effect is to reward the agent for keeping its options open: an action the policy rarely takes gets a bonus, discouraging the policy from collapsing onto a single action too quickly. This is the mechanism that keeps the agent exploring.

To evaluate the entropy terms, write

Then

The advantage-like terms at the four state-action pairs are therefore:

The raw actor drift. The raw drift is

Since , the entropy term simplifies to . Grouping the four terms by their advantage-like factor, and using the occupancy formulas from Section 0.5, gives

Both brackets simplify because the occupancy weights for matching pairs sum to and for non-matching pairs sum to , independently of . Continuing:

This is the key formula for the symmetric example. The raw actor drift depends on and but not on . The cancellation is not a general feature of the model; it is a consequence of the antisymmetric feature structure. We will break it in Section 0.15.

Let us read the three factors of separately, because each one carries a distinct piece of the dynamics.

The factor is the policy’s exploration variance. It is maximal when (uniform policy, ) and it vanishes as the policy becomes deterministic ( or ). This is the softmax analogue of a familiar fact: a deterministic policy has zero score and therefore receives no gradient signal. A platform that has already fully committed to one recommendation strategy has no room to learn from the reward it observes. The exploration variance imposes a built-in speed limit: the more the agent has already learned, the slower it adjusts.

The factor is the advantage-like signal. It is positive when the critic’s value estimate exceeds twice the current actor parameter, which happens when is not yet large enough to justify the reward the critic is reporting. It is negative when has overshot — when the policy is more specialized than the critic’s estimate warrants. The zero crossing will turn out to define the interior equilibrium of the actor.

The drift therefore pushes in the direction that aligns the policy with the critic’s assessment, but the push weakens as the policy becomes more certain. The system cannot accelerate indefinitely: high specialization suppresses the gradient, and the actor slows down before it overshoots.

Boundary damping. The effective actor drift includes a confinement factor that keeps the parameter inside the box :

The factor vanishes on the boundary and is strictly positive in the interior. Its effect is to damp the drift as the parameter approaches the boundary, ensuring that no trajectory can leave the box. This confinement is not a physical constraint on the agent — it is a modeling device that keeps the parameter in a compact set, which is the precondition for the attractor theory developed in later chapters.

The state distribution evolves by relaxation toward a prescribed policy-dependent target law . For this example, define

In the recommendation system, this map says: when the platform uses policy parameter , the long-run fraction of type-1 users is . The linear dependence reflects a mild population response — as the platform specializes toward matching (increasing ), it attracts more repeat visits from the user type that benefits most from accurate recommendations, tilting the population toward state 1. The coefficient keeps this response moderate: even at the extreme , the population split is 75/25, not 100/0. We prescribe as data in this chapter; the identification of with the invariant law of a controlled Markov chain is a separate theorem, deferred to Chapter 6.

Since , the first component lies in , so belongs to the interior of the simplex for every . Each component of is Lipschitz with constant ; in the -distance on , the standing metric for distributions from Chapter 2 onward, the two component changes add, so the Lipschitz constant of is .

The scalar distribution equation is

where we have set the relaxation rate . For frozen , this is a linear equation that drives exponentially toward the target .

For now, is simply a Lipschitz function from the actor box to the simplex, and the distribution equation relaxes toward it.

Collecting the three components, the full coupled system in our example is the three-dimensional autonomous ODE with component equations

The phase space is

In the symmetric case, the subsystem is autonomous: the first two equations do not involve . The population coordinate is still part of the full state, but in this special reduction it responds to without feeding back into the first two equations. This decoupling is accidental and should not be expected in general.

The system is autonomous, and the only irregularity is at the boundary , which is precisely the confinement mechanism we want.

Let us read the system once more in the language of the recommendation platform. The first equation says: the platform adjusts its recommendation strategy based on how the reward signal () compares to the current level of specialization (), but the adjustment weakens as the strategy becomes more decisive (the factor) and as it nears the boundary of the allowed range (the factor). The second equation says: the scorecard updates to track the reward that the current policy actually generates, relaxing exponentially toward its target. The third equation says: the user population drifts toward the mix that the current strategy induces, with the closure map prescribing that target mix. All three quantities from Section 0.1 are now present in equations, but one feedback arrow has disappeared in the symmetric reduction: the policy still drives the population, while the population does not feed back into the actor or critic until we break the symmetry in Section 0.15.

Before we study equilibria or long-run behavior, we need to verify that the system is well-posed on its intended domain. The actor parameter should stay in its box, the distribution should remain a probability, and the critic should not blow up. These properties are not automatic — each requires a separate argument. The three checks below are the concrete version of the forward-invariance arguments that Chapter 4 will develop in full generality.

The phase space is not all of , so we check each component separately.

The actor box. The factor vanishes at . If a trajectory starts with , set ; then , a bounded continuous quantity along the trajectory, so stays finite on every finite interval and therefore for all forward time. If , then and uniqueness of solutions forces for all ; if , the same uniqueness argument gives for all . Chapter 4 carries this barrier argument in the general setting. The mechanism is that acts as a barrier that makes the boundary invariant.

The simplex. The distribution equation has, by variation of constants, the explicit solution

The weights in this expression sum to :

Since they are nonnegative, is a convex combination of and the values , so remains in for all .

The critic. The critic equation satisfies

using and . The function therefore satisfies , and since whenever , the level set is forward invariant for any . We fix the critic radius once and for all; Chapter 2 derives this choice from the general formula , where bounds the critic forcing — here , so . The ball is therefore forward invariant. In practice, the critic relaxes much faster: the equilibrium lies in , and any trajectory with large is pulled back exponentially.

Forward invariance tells us that trajectories stay in the phase space, but it does not tell us they are eventually confined to a small region. An absorbing set is a compact region that every trajectory eventually enters and never leaves. Its existence is the key prerequisite for the global attractor: once a flow has a compact absorbing set, the attractor construction follows from general dynamical-systems theory (Chapter 1 for the definitions, Chapter 5 for the construction). In our three-dimensional system, the absorbing set turns out to be the box defined below.

The compact set

is forward invariant by the analysis above. Moreover, it absorbs every bounded subset of : if the initial critic value satisfies , the exponential decay — obtained by comparing with the scalar ODE (using and rearranging) — guarantees that after a finite time depending on : the asymptotic level sits strictly below , so choosing with gives , hence , for all . (The actor and distribution components are already confined.) In the language of dynamical systems, is a compact absorbing set.

The existence of a compact absorbing set is the main structural prerequisite for the global attractor theorem. In finite dimensions, once the semiflow has a compact absorbing set, the omega-limit set of that absorber is a compact invariant set that attracts every bounded set. That set is the global attractor. In later chapters, we will prove this construction in full generality. Here, we note that the example satisfies the necessary boundedness property.

An equilibrium of the system is a point at which all three components of the vector field vanish simultaneously.

The distribution equation gives .

The critic equation gives .

The actor equation gives . This vanishes if and only if at least one of the three factors is zero:

1. : the actor sits on the boundary, .
2. : impossible, because for every finite .
3. : the actor drift vanishes at an interior point.

Boundary equilibria. At :

At :

In behavioral terms, the equilibrium at is a strong specialist: the platform assigns approximately 88% probability to the matching action at each state, and the user population is 75% type 1. The equilibrium at is the mirror image — an anti-specialist that recommends the wrong content type 88% of the time, with the population tilted the other way.

These boundary equilibria exist because the damping factor vanishes at . They are equilibria of the damped system, not of the raw actor field . In general, , so the boundary equilibria should not be confused with stationary points of an undamped policy gradient. At the raw drift is in fact negative: , because the entropy penalty (the term in the advantage-like signal) outweighs the reward-plus-critic signal at the wall. If the damping were removed, the agent at would retreat toward the interior rather than push further out; it is the confinement factor , not a balance in the reward signal, that makes the boundary points equilibria. Exercise 0.6 carries out this computation.

Interior equilibrium. The third factor vanishes when . Combined with the critic equilibrium condition , the interior equilibrium solves

At , the left side is and the right side is , so the difference is positive. At , the left side is approximately and the right side is , so the difference is negative. By the intermediate value theorem, there is a root in . Numerical computation gives

At this equilibrium, the actor is in the interior of its box, the critic is at its frozen equilibrium, and the distribution is at its target.

In the recommendation system, describes a platform that has settled on a moderate level of specialization: it assigns about 81% probability to the matching content type, leaving roughly one recommendation in five “exploratory.” The critic estimate reflects the moderate reward this strategy earns. Unlike the boundary equilibria, this point is a genuine zero of the raw drift : the advantage-like signal vanishes, meaning the critic’s assessment and the policy’s commitment are in balance. This is the equilibrium where the gradient itself says “stop adjusting,” not where the box wall says “you cannot go further.”

Whether this equilibrium is stable, and how it relates to the boundary equilibria, are questions that the general attractor theory will help answer.

The equilibrium analysis tells us where the system can rest; the phase portrait organizes the directions of motion between those equilibria. Because the subsystem is autonomous in the symmetric case, we can read the essential geometry in the plane and then recover from its linear relaxation equation.

Two curves organize the picture. The first is the critic nullcline , the set of points where . The second is the actor nullcline , the set of points where the advantage-like signal vanishes and . These two curves intersect at the interior equilibrium .

Above the actor nullcline (where ), the raw drift is positive, so the actor parameter increases. Below the nullcline (), the drift is negative and the actor retreats. Inside the actor box, the damping factor is positive, so has the same sign as . Relative to the critic nullcline, the sign is just as simple: below and above it. The critic therefore points toward its nullcline from both sides, while the actor points right above its own nullcline and left below it.

These sign rules already explain why the interior intersection is the natural candidate for the dominant attractor of the planar subsystem. A short numerical experiment is consistent with that picture: representative trajectories in the absorbing set bend toward the interior equilibrium rather than toward the boundary points. Exercise 0.9 asks the reader to reproduce that phase portrait. What Chapter 0 does not do is prove global convergence or classify the boundary equilibria rigorously; that stability analysis belongs later.

The distribution coordinate adds no new planar geometry in the symmetric case. Once is known, follows from the forced linear equation . The graph is therefore the distribution nullcline: it marks the instantaneous target toward which is pulled, rather than an additional autonomous source of recurrence.

The repository file submission/rebuild_attractor_figure.py generates a phase portrait and time-series plot for a closely related asymmetric variant of this system. Exercise 0.9 asks the reader to adapt that script to the symmetric case.

The decoupling of from the subsystem was a consequence of the antisymmetric feature structure combined with symmetric rewards. To see the full coupling, change the reward at state from to for some . In the recommendation system, this corresponds to one user type being more or less valuable than the other: a type-2 match now generates engagement instead of .

With this modification, the critic coefficient becomes

while is unchanged. The critic now depends explicitly on whenever .

The raw actor drift becomes

The term is the coupling that was invisible in the symmetric case. When , increasing (more mass on state ) increases the actor drift, because the higher-reward state-action pair receives more weight. Conversely, when , the coupling reverses sign.

The full asymmetric system is determined by the three component equations

Now the three equations are genuinely coupled: enters all three, enters the first, and enters the first two. No two-dimensional reduction is available without further analysis. This is the generic situation, and it is why the enlarged state space is needed.

The structural properties still hold, with one constant to recheck. The actor box and the simplex are unchanged by the reward modification, because they depend only on the confinement factor and the convexity of ; the coercivity bound is also untouched. The critic forcing, however, is now , and the bracket is a convex combination of (at ) and (at ), so the forcing bound becomes and the critic radius becomes . For — including the value of Exercise 0.5 — this gives and , so the compact absorbing set is the same . For the critic interval enlarges to , and the global attractor theorem will apply to the asymmetric system by exactly the same argument.

The terms used in this section — global attractor, omega-limit set, complete bounded trajectory — will be defined precisely in Chapter 1. Here we use them informally to describe what the example’s long-time behavior looks like.

For a reader coming from reinforcement learning, the usual asymptotic question is: “does the algorithm converge, and if so, to what?” The global attractor generalizes that question. It captures the full set of long-run behaviors the system can exhibit, including cases where the outcome depends on initial conditions or where connecting orbits join multiple equilibria. If the attractor is a single point, the system converges in the usual sense. If it is richer — containing several equilibria and orbits between them — the long-run behavior is richer, and a single convergence statement is too narrow. The attractor is the sharpest answer to “what can happen eventually?”

When Chapter 1 defines the global attractor precisely, it will collect the full set of long-run behaviors in one compact invariant object. The Chapter 0 computations already show some of the ingredients that object must reflect: the compact absorbing box , the three equilibria found in Section 0.13, and the planar sign structure described in Section 0.14.

Since, in the symmetric reduction, the subsystem is autonomous and two-dimensional, the Poincare-Bendixson theorem (limits of bounded planar trajectories are equilibria, periodic orbits, or connections between them; see Coddington and Levinson, Chapter 16) sharply restricts its long-time behavior. No such planar reduction is available for the asymmetric system of Section 0.15, where feeds back into the first two equations. Chapter 0 does not decide which of those possibilities actually occur for this parameter choice. It only places the example in a setting where those alternatives can be stated concretely.

On the distribution side, does not generate independent long-time behavior. It solves a forced linear equation driven by , so any complete bounded trajectory of the full system has its third coordinate tied to the history of the coordinates. The graph remains the natural organizing nullcline for that coordinate.

The purpose of this section is therefore modest. The worked example is already rich enough to display several equilibrium candidates, a compact region that contains all long-time behavior, and a nontrivial distinction between the full three-dimensional system and the planar subsystem that organizes its symmetric reduction.

This chapter has built a complete explicit instance of the model with a prescribed closure map:

• a two-state, two-action environment with scalar actor, scalar critic, and scalar distribution coordinate;
• a softmax policy reducing to the logistic sigmoid;
• a uniformly coercive linear critic equation;
• an actor drift with boundary damping that confines the parameter to a box;
• a distribution equation relaxing toward a prescribed target law.

The resulting three-dimensional autonomous ODE has a compact absorbing set, at least three equilibria (two on the boundary of the actor box, one in the interior), and structural properties–box invariance, simplex invariance, critic coercivity–that guarantee global existence and boundedness. Along the way, the chapter has traced the behavioral meaning of each component: what the agent does at each equilibrium, why the drift slows near certainty, and how the opening three-quantity story becomes a concrete ODE, first in a symmetric reduction and then in the fully coupled asymmetric variant.

What the chapter has not done is prove these properties in full generality, define the asymptotic vocabulary precisely, or construct the global attractor as a theorem. Those are the tasks of the chapters that follow. Chapter 1 will introduce the bridge between RL and dynamical-systems language: semiflows, absorbing sets, omega-limit sets, and the role of the enlarged state space. Chapters 2 through 5 will prove the general attractor theorem for the prescribed-closure model and return to the present example to verify each step.

Exercise 0.1 (Policy computation). Verify that satisfies . Use this to show that the score functions in Section 0.8 are correct.

Exercise 0.2 (Critic equilibrium). Show that for each fixed , the critic equation has exactly one equilibrium, and that every solution converges to it exponentially with rate . Compute the equilibrium explicitly when .

Exercise 0.3 (Simplex invariance by explicit formula). Using the variation of constants formula for in Section 0.11, verify directly that for all whenever and .

Exercise 0.4 (Interior equilibrium equation). Define . Show that and . Conclude that the system has an interior equilibrium. Estimate to two decimal places by evaluating at a few points.

Exercise 0.5 (Breaking symmetry). Set and recompute the critic coefficient and the raw actor drift . Verify that both now depend explicitly on . Find the equilibrium conditions for the asymmetric system.

Exercise 0.6 (Boundary equilibria are not undamped stationary points). At the boundary equilibrium , compute . Verify that it is nonzero and determine its sign. Explain in one sentence why is still an equilibrium of the full system despite .

Exercise 0.7 (Effect of the closure map). Replace the closure map with the constant map . How does this change the equilibrium structure? Does the absorbing set change?

Exercise 0.8 (Absorbing set entry time). Suppose and , . Using the exponential decay bound from Section 0.12, estimate the time after which .

Exercise 0.9 (Phase portrait). Implement a numerical integrator (e.g., fourth-order Runge-Kutta) for the symmetric system of Section 0.10. Plot trajectories in the plane from three initial conditions: , , and . Verify that all three converge to the interior equilibrium. On the same plot, draw the actor nullcline and the critic equilibrium curve , and confirm that they intersect at . The repository file submission/rebuild_attractor_figure.py implements a related plot for the asymmetric variant and can serve as a starting point.