Chapter 2: The General Model

Chapter 1 ended with a program: starting from the semiflow, absorbing set, and global attractor vocabulary, prove that a general finite-state actor-critic mean-field system has a compact global attractor. In Chapter 0, that system appeared as a recommendation platform whose strategy, scorecard, and audience mix evolved together, and the three scalar variables were enough to see the feedback loops, the confinement, and the absorbing set by hand. The gap is that the general model–with states, actions, a -dimensional actor parameter, and an -dimensional critic–has not yet been written down. Until it is, we cannot state the theorems we intend to prove.

This chapter fills that gap. We define each component of the model separately–the softmax policy, the generator family, the product occupancy, the linear critic, and the damped actor drift–and then assemble them into a single autonomous ODE on the enlarged phase space introduced in Section 2.6. For each definition we verify that the general formula recovers the Chapter 0 special case, so that the general system, specialized to and , reproduces the Chapter 0 equations line by line. Because the chapter is definitional, the arguments here are computations rather than theorems; well-posedness and the a priori estimates begin in Chapters 3-4.

In Chapter 0, the policy was a logistic sigmoid that determined which content the recommendation platform promoted to each user type: a single scalar controlled the split between two actions at two states, and the antisymmetric feature structure made the sigmoid appear naturally. The general version is a softmax that converts a parameter vector into a probability distribution over arbitrarily many actions at each state.

Fix integers , , , and , and fix a box radius . Let be the state space, the action space, and the actor box used later in the prescribed-closure phase space. Fix a temperature parameter and actor feature vectors for each .

Definition 2.1 (Softmax policy). For , , and , define the logit

and the softmax policy

At each state , the softmax produces a probability distribution over : the weights are nonnegative and sum to . The temperature controls how sharply the policy concentrates on the action with the largest logit, so that as the policy becomes deterministic (argmax), and as it approaches the uniform distribution over actions. For any finite , every action receives strictly positive probability:

Strict positivity is what makes the softmax well-suited to the analysis: the logarithm is always finite, and the score function (defined next) is always well-defined.

The score function. The score function is the gradient of the log-probability with respect to the actor parameter. Differentiating the logarithm of the softmax gives

The expression in parentheses is the difference between the feature vector for action and the policy-weighted average of feature vectors at state , so the score points in the direction that, locally, increases the probability of action relative to the mean action. Its magnitude is bounded:

This bound holds because the bracketed term is a difference of two vectors, each of norm at most .

Verification in the Chapter 0 example. In Chapter 0, we had , , , and the feature scalars

The logits are , , , . The softmax at state gives

recovering the logistic sigmoid from Section 0.4. The antisymmetric feature structure makes the same formula hold at state with the roles of actions swapped.

The score function at state for action is

which agrees with Section 0.8. The state-1 score for action 2 is . The score function bound gives , and indeed for all finite .

Takeaway. The softmax policy converts an actor parameter into a strictly positive probability distribution over actions at each state, and its score function is bounded. These two properties–strict positivity and bounded score–recur in the regularity arguments of Chapter 3 and the a priori estimates of Chapter 4.

The softmax policy determines how the agent acts, but to describe how the state distribution evolves we need a model of the environment’s response. In the continuous-time setting, that model is a family of Markov generators indexed by the policy parameter.

For the RL reader. If you are accustomed to discrete-time transition matrices , the generator plays an analogous role in continuous time. The matrix exponential is the transition matrix for running the chain for time ; the off-diagonal entries of are the instantaneous transition rates; the diagonal entries are chosen so that rows sum to zero (instead of summing to one, as they would for a stochastic matrix). The essential content is the same: the generator encodes which state transitions are possible and how fast they occur.

In the recommendation platform of Chapter 0, the generator would capture how users transition between behavioral types–how a type-1 user, exposed to the platform’s content choices, might shift toward type-2 behavior, and at what rate.

Definition 2.2 (Generator family). For each , let

be a generator matrix, meaning that

The row-sum condition is equivalent to , where . The generator acts on observables (column vectors): if , then is the instantaneous rate of change of along the chain at state . Its transpose acts on distributions (also column vectors): if , then is the instantaneous rate of change of the distribution under the chain.

In the RL setting, the dependence of on the actor parameter comes through the policy: the agent’s action choices at each state determine which transitions occur, and at what rate. The canonical construction is to fix a base transition kernel (the probability that action at state causes a transition to state ) and define

This is the construction the bridge theorem of Chapter 6 uses to identify the prescribed closure with the invariant law of the controlled chain. The same recipe applies when the environment is specified by action-conditioned rate matrices instead of a probability kernel: given generators with off-diagonal rates , set for ; the probability-kernel formula is the special case in which every action jumps at unit rate and lands according to . The three-state routing example of Section 2.8 is a direct instance of this rate-matrix version: the action-indexed generators and carry the rates, and is the softmax-weighted mixture of their rows. Since is smooth in , either construction makes smooth in . What matters for the analysis is that is a generator for each and that the family is regular enough–conditions made precise in Assumption 2.8.

The scaled law equation. Given an actor trajectory , the exact state-distribution equation is

The parameter is a timescale ratio. When is small the distribution evolves rapidly relative to the actor, so that the chain explores its state space quickly compared to the pace of policy updates.

Because for every , the transpose satisfies , which means

In words: the total mass is conserved. If the initial distribution is a probability vector (), the total mass remains for all time. When is frozen, the semigroup maps probability vectors to probability vectors (a finite-state Markov-chain fact proved in Norris, Markov Chains, Chapter 2; we import it with its exact statement where it is first used seriously, in Chapter 6, Section 6.2). For a time-dependent actor path , the natural object is instead the non-autonomous evolution family generated by , which has the same mass-preserving and positivity-preserving structure. So is still the natural state space for the exact law equation, even though the formula is no longer a single matrix exponential. We return to that non-autonomous viewpoint in Chapter 7.

Verification in the Chapter 0 example. For , a generator has the form

where, in the entry notation of Definition 2.2, and are the transition rates from state to state and from state to state , respectively. The row sums vanish, so . The transpose is

The law equation in the scalar coordinate is

For frozen , this is a scalar linear ODE that relaxes exponentially toward the equilibrium value , with rate . Provided both rates are strictly positive, the invariant law is uniquely determined and lives in the interior of the simplex.

In Chapter 0, the distribution equation was , which has the form of a relaxation toward the prescribed target . That equation is not the generator law equation above; it is the prescribed-closure law that replaces the exact chain dynamics with a target map, a replacement we introduce in Section 2.6. The generator formulation here is the exact one, which is needed when the bridge theorem (Chapter 6) connects the prescribed closure to the genuine invariant law of the controlled chain.

Takeaway. The generator family determines how the state distribution evolves in continuous time. The exact law equation preserves the simplex and conserves total mass. In the full coupled system, the actor drives the generator through its dependence on , and the generator drives the distribution, creating a feedback loop.

The actor drift in a policy-gradient method depends on expectations over state-action pairs. To write those expectations cleanly we need a single object that records both where the agent is and what it does, namely the occupancy measure.

Definition 2.3 (Occupancy measure). For , define the state-action occupancy measure

Since and , the occupancy is a probability distribution on :

The occupancy depends on two inputs: the policy parameter , which determines how the agent acts, and the current state distribution , which determines where the agent is. This is the product occupancy introduced in Chapter 0. At this stage it is a derived quantity from the current state variables, not a stationary measure of any Markov process.

The critic equation. The critic variable is an auxiliary state coordinate that feeds into the actor drift. Its dynamics are a linear response law whose coefficients depend on the occupancy.

Fix critic feature vectors for each , a regularization constant , and reward values . (The notation corresponds to the of Chapter 0; we use from now on to match the proposal.)

Definition 2.4 (Critic coefficients). Define the critic forcing vector and matrix:

The critic equation is

The vector is the reward-weighted average of the critic features under the current occupancy: it summarizes, in the critic’s feature coordinates, how much reward the policy is currently earning across all state-action pairs. The matrix is a regularized feature-correlation matrix: its first term provides a baseline contraction that pulls the critic toward zero, and its second term adds the occupancy-weighted outer products of the critic features, capturing how much information the current state-action distribution provides about the value function. The critic equation therefore says: adjust in the direction of the current reward signal , but pull it back by the correlation-plus-regularization matrix , with the balance between these two forces determining the critic’s equilibrium.

For the dynamical-systems reader. The critic variable may look like an unnecessary complication if you are used to ODEs given by a single explicit vector field. In the RL setting, represents a learned value estimate that the actor uses to evaluate its policy. The structural point is that is not a derived quantity that can be eliminated from the system: it evolves on its own timescale, and its transient value affects the actor drift. Dropping from the state would require assuming it has already converged, which is exactly the informal fast-critic substitution that the rigorous analysis must justify rather than assume.

The matrix is uniformly positive definite. Because the first term is and the second term is a sum of positive semidefinite rank-one matrices, we have

for every . Uniform coercivity means that for any frozen the critic equation is a stable linear ODE: every solution converges exponentially to the unique equilibrium

The exponential rate is at least , uniformly in . This is the dissipative mechanism that keeps the critic variable bounded and eventually contributes to the compact absorbing set.

Verification in the Chapter 0 example. In Chapter 0, with , , and the matching-action critic features (, , , ), the general formulas give

which matches Section 0.7. The critic equation reduces to the scalar ODE with equilibrium , exactly as computed in Chapter 0. The coercivity constant is , confirming .

Takeaway. The critic equation is a linear ODE in whose coefficients depend on the occupancy. Uniform coercivity of guarantees exponential relaxation of the critic for any frozen policy and distribution, and this coercivity is the primary dissipative mechanism for the critic component in the full system.

The actor equation is the heart of the model: it updates the policy parameter in a direction determined by the occupancy, the critic estimate, and the policy score. We first define the raw drift, then introduce the boundary damping that confines the actor to a compact box.

Recall that the actor parameter space is

Definition 2.5 (Raw actor drift). For , define

The raw drift is a score-weighted average over all state-action pairs. Each pair contributes in proportion to its occupancy weight , the direction of the contribution is the score function , and the magnitude is modulated by a signal that combines three contributions. We unpack the three contributions in turn.

The immediate reward measures how good action is at state , independent of any learned estimate. In the recommendation platform, this is the raw engagement a content choice generates.

The critic’s value estimate is the learned correction: it adds the critic’s current assessment of how much long-run value action at state delivers, projected onto the critic features. When the critic has converged, this term reflects the true return; during transients, it reflects the critic’s best current guess.

The entropy-regularization bonus is large and positive when the policy assigns low probability to action (since is then a large negative number), and small when the policy already favors . Its effect is to reward the agent for keeping its options open, preventing the policy from collapsing onto a single action before the evidence warrants it.

Together, the three contributions form an entropy-regularized advantage-like signal: the agent is pushed toward actions that earn high reward, that the critic values, and that the policy has not yet committed to. This is the general version of the advantage decomposition in Section 0.8.

The raw drift is well-defined and smooth on all of , since the softmax policy is strictly positive (so the logarithm is finite), smooth in , and affine in and .

Growth estimate (preview). The raw drift grows at most linearly in the critic norm once is restricted to the actor box . Since is compact, the logits and therefore the logarithms are uniformly bounded there. Combined with the bounded score function and the fact that the occupancy is a probability weight, this gives constants , depending only on the finite data, such that

The qualitative point is enough for now: the only unbounded contribution comes from the linear term , which is why the critic coercivity matters. Chapter 3 proves that the actor field is regular enough for the local ODE theory, and Chapter 4 uses the critic bounds to turn this linear growth into global control.

Definition 2.6 (Boundary damping). Define the diagonal damping matrix

The effective (damped) actor drift is

The factor is the confinement mechanism. On the interior of , every diagonal entry is strictly positive, so is positive definite and points in the same direction as . On the boundary, where some coordinate , the corresponding diagonal entry vanishes, from which it follows that the -th component of the drift shuts off. This coordinatewise vanishing is the mechanism behind the forward-invariance proof in Chapter 4: at each face of the box, the corresponding component loses its outward motion instead of pushing the trajectory through the wall.

In one dimension the mechanism is visible from the factorization : near the drift vanishes linearly in , and an arctanh change of variable transforms the confined ODE into one with bounded drift on , so the transformed coordinate cannot reach infinity in finite time. This is the one-dimensional picture behind the barrier argument: the damping slows the motion so strongly near the wall that the boundary is approached asymptotically. Chapter 4 turns that intuition into the formal invariance proof used later.

Verification in the Chapter 0 example. In Chapter 0, and , so and the actor equation is

matching Section 0.8 exactly. The damping vanishes at , and in Chapter 0 this is the mechanism that confines the actor to and creates the two boundary equilibria.

Takeaway. The actor drift combines the policy-gradient direction (the score) with a reward-plus-critic signal, weighted by the occupancy. The boundary damping matrix is the confinement mechanism for the box : it leaves the interior direction unchanged and forces the relevant coordinate drift to vanish at each boundary face. On , the raw drift grows at most linearly in the critic norm .

The model defined in Sections 2.1—2.4 involves several data choices: the state and action spaces, the feature vectors, the reward values, the temperature, the regularization constant, and the generator family. Not all choices lead to a well-behaved system. The standing assumptions collected here are the conditions used in Chapters 3—5.

Assumption 2.7 (Finite-state data). The integers , , , , the constants , , , the reward values , the actor features , and the critic features for are fixed throughout. The actor parameter space is .

This data determines the softmax policy, the occupancy measure, the critic coefficients, and the actor drift as defined in Sections 2.1—2.4. No further conditions on the features or rewards are needed for the prescribed-closure theory: the features may be degenerate, the rewards may be zero, and the policy may be nearly uniform; the estimates in Chapters 3—5 are expressed in terms of finite bounds extracted from this data.

Assumption 2.7 is the data used directly in Chapters 3—5; every estimate there is expressed in terms of the constants it defines. The next assumption adds the regularity of the generator family, which those results do not use but which becomes essential when the prescribed closure is identified with the invariant law of the controlled chain in Chapter 6. We state it here so that the full set of hypotheses is visible in one place.

Assumption 2.8 (Generator regularity, ass:gen). For every , the matrix is a generator:

The map is Lipschitz from into the space of matrices with the induced -operator norm: there exists such that

Here we use the -norm on , , and the induced operator norm . The -norm is natural here because the distribution variable lives in the probability simplex, and the -distance between two probability vectors is twice their total variation distance.

The first condition (generator property) ensures that defines a continuous-time Markov chain on for each frozen ; the second condition (Lipschitz adjoint) ensures that the generator family depends regularly on the policy parameter. Together, they are the structural controlled-chain hypotheses carried into Chapter 6: they make a linear law equation whose coefficients vary regularly with . To obtain a single invariant-law map with quantitative continuity or Lipschitz bounds, Chapter 6 adds a separate uniform exponential mixing hypothesis.

What each condition does:

• Generator property: The off-diagonal non-negativity and row-sum-zero conditions together imply that the matrix exponential is row-stochastic for every . Equivalently, maps into , so the law equation preserves probability vectors.

• Lipschitz adjoint: This controls how the frozen generator changes with . In Chapter 6, once uniform exponential mixing is added, it feeds a resolvent estimate on the zero-mass subspace that makes Lipschitz, with constant proportional to (Theorem 6.4); a separate Doeblin-style minorization criterion (Section 6.5) gives a checkable sufficient condition for the mixing hypothesis itself. By itself the Lipschitz adjoint does not guarantee a unique invariant-law map.

Verification in the Chapter 0 example. The Chapter 0 model did not specify a generator explicitly, since system (L1) uses the prescribed closure map instead. For the 2-state example of Section 2.2, however, we check both conditions directly. If the rates and are smooth positive functions of , then is a generator for each (the off-diagonal entries are positive, the rows sum to zero), and the map is Lipschitz on the compact interval since it is there.

Takeaway. The finite-state data (Assumption 2.7) is needed for the prescribed-closure theory in Chapters 3—5. The generator regularity (Assumption 2.8) is the baseline controlled-chain input for Chapter 6. The additional mixing hypothesis that turns into a well-behaved invariant-law map is deferred to that bridge chapter.

We now assemble the model. The previous sections defined the actor drift, the critic equation, and the exact law equation. To obtain a self-contained autonomous system that the prescribed-closure theory can analyze we need one more ingredient: a prescribed rule that assigns a reference distribution to each policy parameter.

Why the system needs a closure. Recall from Chapter 1 (Section 1.8) that the actor drift depends on the occupancy , which in turn depends on the state distribution . The distribution evolves by its own equation, driven by the generator . In the exact system the actor, critic, and distribution are all coupled. We separate two questions:

1. Does a system with a prescribed Lipschitz closure map have a well-behaved global attractor?
2. Can that prescribed map be identified with the invariant-law map of the controlled chain?

The first question is self-contained ODE theory; the second requires analyzing the controlled Markov chain (Chapter 6). Separating them yields a modular argument: the attractor theorem holds for any Lipschitz closure, and the bridge theorem then identifies the physically relevant one.

For both audiences. The prescribed closure is a modeling choice that affects how evolves, not how the occupancy is built. In system (L1) the occupancy is still formed from the current dynamic distribution , and the actor and critic drifts depend on all three state variables. What changes is the distribution equation: instead of the generator dynamics , the distribution relaxes toward the prescribed target at a fixed rate. This keeps the system fully coupled–all three equations depend on each other–while avoiding the need to analyze the controlled chain until Chapter 6.

Definition 2.9 (Prescribed closure map). Fix a Lipschitz map

The Lipschitz constant of , measured on with the -norm on , is denoted .

The map is required to take values in the simplex , so is a probability vector for every . For ambient ODE arguments in Chapter 3, we use the coordinatewise clipping map and the extension This extension is globally Lipschitz with the same constant because is -Lipschitz, and it agrees with on . In the exact system (Chapter 6), is identified with the frozen invariant-law map . In the prescribed-closure system it is data.

Definition 2.10 (Prescribed-closure dynamical system). The prescribed-closure mean-field flow is the autonomous ODE

where is a relaxation rate. The phase space is

We equip with the product metric

where the -component carries the -norm of Section 2.5–the same norm in which the closure constant of Definition 2.9 is measured.

Several features of this system deserve comment.

First, the system is fully coupled: the actor drift depends on , the critic drift depends on , and the distribution equation depends on . No two-dimensional reduction is available in general. The mechanism is direct: the occupancy carries linearly into every formula in Sections 2.3 and 2.4, so the critic forcing is a sum of -weighted reward contributions, and the raw actor drift is a sum of -weighted score contributions. The Section 2.8 routing example shows this concretely: the actor drift contains the combination , which varies with the state law and cannot be replaced by a function of alone. In the symmetric Chapter 0 example, the occupancy-weighted sums happened to cancel the -dependence in the actor and critic equations, producing an autonomous subsystem–an artifact of the antisymmetric feature structure, not a general property. In the asymmetric variant (Section 0.15) the full coupling was already visible.

Second, the distribution equation is a linear relaxation toward . For any frozen it drives exponentially toward with rate . Since and the equation is affine, the simplex is forward invariant: if , then for all .

Third, the full system is autonomous: the right-hand side depends on and on the fixed data, but not on time. This is a consequence of working in a stationary environment.

We read the three equations in plain language. The first equation updates the policy parameter : the actor moves in the direction indicated by the policy gradient, weighted by the occupancy and modulated by the reward-plus-critic signal, while the boundary damping decelerates the motion as approaches the walls of the box . The second equation adjusts the critic’s running estimate : the forcing pushes toward the reward signal, while the matrix pulls it back, and the balance between push and pull determines where the critic settles for any given policy and distribution. The third equation relaxes the state distribution toward the prescribed target at rate : when the policy changes the target shifts, and the distribution follows with exponential tracking. The three equations close the feedback loop that Chapter 0 described for the recommendation platform: the strategy influences the audience mix, the audience mix influences the reward signal, and the reward signal influences the next strategy revision.

The exact system. For reference, the exact finite-state system with the generator dynamics is

The first two equations are identical to system (L1)–the actor and critic drifts depend on the dynamic in both cases. What differs is the third equation: in the exact system evolves by the generator , while in the prescribed-closure system it relaxes toward . The exact system therefore fits the same enlarged-state framework but requires controlled-chain estimates beyond the prescribed-closure theory; those estimates are developed in Chapters 6 and 7. In the companion short note, the label eq:main names this exact generator system, and the attractor theorem there (Theorem 2) is stated directly for it; the prescribed-closure relaxation above therefore carries the separate tag (L1).

Standing constants. For system (L1), the following constants are determined by the data and appear in the estimates of Chapters 3—5:

The constant is the radius of the critic absorbing ball.

Each constant has a concrete reading. The reward bound caps how much reward any single state-action pair can deliver. The actor-feature bound caps how sensitive the policy logits are to a unit change in , so that a large means the policy can swing sharply with small parameter adjustments. The critic-feature bound caps how much a single state-action pair can influence the critic’s estimate. The product bounds the magnitude of the critic forcing vector : it is the worst-case rate at which reward information enters the critic. Finally, is the radius beyond which the critic’s own dissipation (the coercivity of ) dominates the forcing, pulling back toward the origin; any trajectory with sees its critic norm decrease. In the Chapter 0 example, all feature and reward bounds equal , so and .

Verification in the Chapter 0 example. In Chapter 0, the full system was

The closure map was , which is Lipschitz with constant in the -norm on : each component moves by , and the two contributions add. (In the scalar -coordinate used in Chapter 0 the constant is .) The relaxation rate was . In the general formulation the actor and critic drifts depend on through the occupancy; in the symmetric Chapter 0 example the antisymmetric feature structure causes the -dependence to cancel, so that and depend on and alone. This is the special cancellation noted in Section 0.8, not a general property of system (L1). In the asymmetric variant (Section 0.15), the full coupling was visible. The phase space was , and the standing constants were , , , , .

Takeaway. The prescribed-closure system is an autonomous ODE on the enlarged phase space . It uses a prescribed Lipschitz closure map to specify the target of the distribution equation while leaving the occupancy equal to . The exact system, which replaces the relaxation by the generator equation, is fully coupled and will be analyzed separately.

We now verify that the general formulation of Section 2.6, specialized to the Chapter 0 data, reproduces the Chapter 0 system exactly. This is a line-by-line check that connects the abstract notation to the concrete formulas the reader already knows.

Data. Set , , , , , , , . Use the Chapter 0 features and rewards:

• For , use , , and .
• For , use , , and .
• For , use , , and .
• For , use , , and .

Set the closure map .

Policy. Definition 2.1 gives logits , , , , and the softmax at state reduces to , recovering Section 0.4.

Occupancy. The general occupancy is , which depends on . In the Chapter 0 feature structure the occupancy-weighted sums that appear in the critic and actor drifts simplify: matching-action terms sum to and non-matching terms sum to , independently of . This cancellation is the antisymmetric feature accident from Section 0.8.

Critic. Because of the cancellation above, Definition 2.4 gives and –both independent of . The critic equation is , exactly Section 0.7.

Actor. Definition 2.5 gives the raw drift. Using the Chapter 0 score functions ( for the matching action, for the non-matching action) and the advantage-like terms from Section 0.8, the computation yields , again independent of . The damping matrix is , so the effective drift is , exactly Section 0.8. In the asymmetric variant (Section 0.15), the -dependence reappears.

Distribution. The relaxation equation is , exactly Section 0.9.

Phase space and constants. The phase space is , and the standing constants are , , , , . The compact set is the candidate absorbing set, matching Section 0.12.

Every formula agrees. The general model is a faithful extension of the worked example.

The Chapter 0 system was chosen because every formula collapses to a scalar and the geometry can be drawn by hand, a simplicity that is pedagogically valuable. We now move to a slightly larger exact actor-critic system that is closer to a real business-routing decision: an agent decides how strongly a platform should route users from ordinary retail browsing toward a veterinary service funnel. The example shows, in explicit data, what the definitions of Sections 2.1-2.6 look like when the state law is a genuine controlled-chain variable rather than a prescribed closure.

We keep the actor and critic scalar:

The state space has three states:

• core retail
• health discovery
• vet booking

The interpretation is the following. State is the ordinary commerce hub: food, supplies, repeat purchases, and the rest of the retail journey. State is a health-information layer: symptom checkers, advice pages, or other content that moves the user away from the shopping flow and toward service consideration. State is the booking funnel itself. The actor parameter controls how aggressively the interface pushes users toward the service layer.

We use two actions at every state:

• retail-safe presentation
• vet-push presentation

The actor features are state-independent:

Definition 2.1 then gives

for each state . Positive means the platform leans toward the vet-push action; negative means it backs off and preserves the retail flow.

The rewards are chosen so that the retail-safe action is best in the retail hub, while the vet-push action is best deeper in the service funnel:

• Core retail (): and .
• Health discovery (): and .
• Vet booking (): and .

These numbers are not meant to be calibrated business values. They are a clean way to encode the qualitative fact that the booking funnel is valuable only if the user is already far enough along the service journey, whereas pushing vet booking too early is counterproductive in the retail hub.

For the critic, we choose

With this choice, the scalar critic measures the running preference for the vet-push side over the retail-safe side. Definition 2.4 gives

so the critic coefficient is constant. The forcing term is

that is,

The critic equation therefore reads

The actor equation is equally explicit. Since and , the raw actor drift from Definition 2.5 simplifies to

For the chosen rewards,

so

and the damped actor equation is

This is the main structural point of the example. The actor drift is not a function of alone; it depends on the current state law through the weighted combination . When most users are in the retail hub the distribution term pushes away from aggressive vet routing; when the population has already moved into health discovery or booking, the same term pushes in the opposite direction. This is the kind of feedback that disappears if one tries to close the actor equation on alone.

To define the law equation, fix two action-conditioned generators. Under the retail-safe action , take

and under the vet-push action , take

Each row sums to zero, and all off-diagonal entries are nonnegative, so both are valid generators. The policy-dependent generator is the softmax mixture

Writing out the off-diagonal rates gives

with all other off-diagonal rates equal to , so that the law equation is

In coordinates,

with . This is the first time we see the exact generator equation instantiated with numerical data. In every earlier computation the distribution equation used the prescribed closure; here the chain dynamics are explicit, and the invariant law is no longer given by fiat but must be computed from .

This example already exhibits two features that later chapters isolate as theorems. First, the policy enters the chain through a genuine softmax actor, not through a direct insertion of into the rates. Second, the retail hub state remains uniformly reachable:

A uniform lower bound on return-to-hub rates is what Chapter 6 needs to obtain a single Lipschitz invariant-law map with quantitative mixing rate, which in turn feeds the singular-limit tracking estimate of Chapter 7. In the language of Chapter 6, this is the beginning of a reference-state minorization condition, with the retail hub playing the role of the reference state (denoted in Chapter 6’s notation). We do not use that theorem here, since Chapter 2 is only building the model. The point is simpler: the abstract ingredients from Sections 2.1-2.6 are already flexible enough to encode a realistic routing problem in which actor, critic, and state law all remain visibly coupled.

This chapter has formulated the general finite-state actor-critic mean-field system. The ingredients are:

• a softmax policy with bounded score function, defined by actor features on a finite state-action space (Section 2.1);
• a generator family that determines how the state distribution evolves under the controlled chain (Section 2.2);
• a product occupancy measure and a uniformly coercive linear critic equation (Section 2.3);
• a damped actor drift that confines the policy parameter to a compact box (Section 2.4);
• standing assumptions on the data and the generator regularity (Section 2.5);
• a prescribed Lipschitz closure map that determines the target of the prescribed-closure distribution equation while leaving the occupancy equal to (Section 2.6).

The full prescribed-closure system is an autonomous ODE on the enlarged phase space . We verified in Section 2.7 that it recovers the Chapter 0 example exactly, and Section 2.8 then showed that the same formal definitions can also carry a more realistic three-state routing problem in which the policy acts through action selection, the controlled chain is exact, and the actor drift depends genuinely on the current state law.

In the general model the phase space has dimension , and the phase portrait is no longer a planar picture. When the system is specialized to the Chapter 0 data, however, the phase space collapses to , and the portrait of Section 0.14–with its interior equilibrium, boundary equilibria, and critic nullcline–is recovered exactly.

What the chapter has not done is prove that this system is well-posed (Lipschitz vector field, global existence, uniqueness) or that it possesses the dissipative properties needed for the attractor theorem (forward invariance, absorbing set, asymptotic compactness). Those are the tasks of Chapters 3 and 4. We now have a complete, precisely stated system and can follow the estimates and constructions that begin next.

Exercise 2.1 (Score function computation). Compute the score function for the Chapter 0 features (, , , ) using the general formula from Section 2.1. Verify that the results match the explicit score functions in Section 0.8.

Exercise 2.2 (Generator matrix). Write out the generator matrix for a two-state controlled chain with transition rates and , for . Verify that the off-diagonal entries are nonnegative and that rows sum to zero. Compute the invariant law and confirm that it equals . Compute the Lipschitz constant of on .

Exercise 2.3 (Critic equation verification). Show that the general critic equation of Definition 2.4, specialized to the Chapter 0 data, gives . Verify that the coercivity bound holds with .

Exercise 2.4 (Lipschitz constant of the closure map). For the Chapter 0 closure map , compute the Lipschitz constant with respect to the -norm on . Verify that for all .

Exercise 2.5 (What happens without the damping). Remove the damping function by setting for all . In the Chapter 0 model, consider the resulting actor equation on . Show that the boundary equilibria at disappear. Is the box still forward invariant? Describe what happens to a trajectory starting at with at its equilibrium value.

Exercise 2.6 (Closure is essential). Explain, in one paragraph, why the actor equation cannot be written as a self-contained ODE in the general model. Identify the specific term that requires knowledge of the state distribution or a closure assumption. State precisely what replaces.

Exercise 2.7 (Assembling a larger system). Write down system (L1) for , , , with generic features and . Do not solve the system. Identify the dimensions of the phase space and count the number of scalar ODEs. Specify the shape of the damping matrix , the critic matrix , and the closure map .

Exercise 2.8 (Retail-to-vet routing preview). In the three-state routing example of Section 2.8, verify directly that and are generators and that is therefore a generator for every . Then check the lower bounds and . Explain in one paragraph why these bounds mean that the retail hub remains uniformly reachable, regardless of how aggressively the actor pushes the vet service.