Chapter 9: From Theory to Models

The main theory is now complete. Chapters 2—5 proved that a general finite-state actor-critic mean-field system defines a continuous semiflow on with a compact global attractor. Chapter 6 supplied the controlled-chain closure by proving that the frozen invariant-law map is Lipschitz under uniform exponential mixing. Chapter 7 turned that closure into a finite-time tracking estimate and upper semicontinuity of attractors as .

Throughout that development, two running examples carried the theory: the Chapter 0 two-state recommendation model and the three-state routing chain from Chapter 2, Section 2.8. Both were introduced as given data (states, actions, features, rewards, generators), and the reader’s task was to verify that the abstract theory specialized correctly. This is the working mode for a theory-first presentation, and it leaves a gap: the reader who wants to model a new domain has no systematic procedure for filling the Chapter 2 template.

This chapter fills that gap. It takes the Chapter 2 template apart, identifies the design decisions that determine the coupled system’s structure, and shows how domain knowledge guides each decision. The result is a six-step protocol that translates a domain problem into a specific ODE on the enlarged phase space. We illustrate the protocol by re-deriving the two known models, and then use the analysis to explain connections between the modeler’s choices and the attractor geometry that the theory reveals.

The prescribed-closure system defined in Chapter 2 is not a single dynamical system. It is a parametric family of dynamical systems, indexed by a finite collection of data:

with the final pair furnished either directly, as a prescribed Lipschitz closure map with a relaxation rate , or replaced by a generator family with a timescale in the exact-chain presentation of Section 2.2 and Chapter 7. Section 9.2, Step 6 develops both presentations.

Each choice of data produces a specific vector field on a specific phase space and therefore a specific semiflow with its own attractor geometry. The state space determines the dimension of the simplex. The features and shape the coupling between the three equations. The rewards set the scale and direction of the actor drift. The temperature and the regularization control the sharpness of the policy and the coercivity of the critic. The closure component (or the underlying generator family ) fixes how the population responds to the policy, and the relaxation rate (or, in the controlled-chain presentation, the timescale ) sets how fast that response unfolds.

When we say “instantiate the Chapter 2 template for a specific domain,” we mean: choose the data so that the resulting ODE captures the feedback loop of interest, and then apply the theorems of Chapters 3—7 to analyze the system’s long-time behavior.

The Chapter 0 model was one instantiation: , , antisymmetric actor features, matching critic features, and a linear closure map relaxed at unit rate (). The Section 2.8 routing chain was another: , , , state-independent features, and an explicit generator family built from the retail-vet topology. Both sets of data were presented as given and then verified against the general formulas. The reader learned what “checking the general case against the special case” looks like, but not what “choosing the data from domain considerations” looks like.

Let us now re-read the Chapter 0 model as a sequence of design decisions, each driven by a domain question.

The platform recommends content to two user types. That gives two states. The platform can promote one of two content strategies. That gives two actions. The actor parameter is scalar because a single number controls the split between the two strategies, the simplest policy parametrization whose gradient does not vanish identically. The features , , , encode an antisymmetry hypothesis: action 1 is preferred at state 1 and action 2 is preferred at state 2, with equal strength. The rewards , , , say that matching the action to the user type is rewarding. The critic features track which action-state matches occur. The closure map says that a policy that specializes toward matching (larger ) attracts more type-1 users.

Each choice corresponds to a specific modeling decision guided by a domain picture: two symmetric user types, two matching content strategies, a linear audience response. A different domain picture would produce different data.

We now formalize the translation from domain to data as a six-step protocol. Each step asks a domain question and produces a component of the data tuple .

Step 1. State space from domain dynamics. Identify the situations the environment can be in, where each situation leads to different agent behavior and different rewards. Each distinct situation is a state.

The key judgment is the level of granularity. Two states that differ in name but not in the agent’s interaction should be merged. Two states that the agent treats identically but that lead to different transitions should be kept separate.

In Chapter 0, there are two user types: the platform interacts differently with each. In Section 2.8, there are three operational states (retail hub, health discovery, vet booking) through which customers flow.

Step 2. Action space from agent capabilities. List the distinct behavioral modes the agent can select at each step. Actions should differ in their effect on the environment’s transitions or on the rewards, beyond a difference in label.

In Chapter 0, the two actions are two content strategies. In Section 2.8, the two actions are two presentation modes (retail-safe, vet-push) that change the customer’s transition probabilities between states.

Step 3. Actor features from policy degrees of freedom. Choose the actor dimension and the feature vectors . The feature vector determines the logit , which controls how the policy probability responds to the parameter .

The features enter the raw actor drift (Definition 2.5) through the score function of Section 2.1:

The dimension is the number of independent policy axes. A scalar actor () means the policy has one degree of freedom that controls all state-action probabilities simultaneously. A vector actor () allows independent control of different aspects of the policy.

The feature structure encodes a symmetry hypothesis. This is the most consequential design choice in the protocol, and we analyze it in detail in Section 9.3.

Step 4. Critic features from reward-relevant distinctions. Choose the critic dimension and the feature vectors . The critic features determine the matrices and vectors that define the critic equation. Recalling the Chapter 2 definitions,

so the critic features control the curvature of the regression problem and the direction of the reward projection.

Choosing aligned with (often with ) makes the critic estimate the reward signal in the same coordinates the actor uses to adjust the policy.

Step 5. Rewards from domain objectives. Assign a reward to each state-action pair. The reward table is the modeler’s assertion about what outcomes are valuable. In RL applications, rewards come from the domain: click rates, revenues, health outcomes, service-level metrics.

The reward table interacts with the features to determine the actor drift. Changing a reward changes the direction of the gradient and hence the location and character of the equilibria.

Step 6. Generator family from transition structure. For controlled-chain models, construct a family of generators that encode how the environment’s state transitions respond to the agent’s policy. The construction proceeds in three parts:

(a) Fix action-conditioned base generators for each action . Each is an matrix with nonnegative off-diagonal entries and zero row sums, encoding the transition rates when the agent commits to action at every state.

(b) Form the softmax mixture: for each pair of states , the effective transition rate from to is

(c) Verify the generator property of and compute its Lipschitz constant in .

For prescribed-closure models, the modeler may instead supply a prescribed Lipschitz closure map directly, identifying it from domain reasoning rather than from a generator, together with a relaxation rate for the distribution equation of Definition 2.10. The Chapter 6 bridge theorem (Corollary 6.5) ties the two presentations together: under the Chapter 6 uniform mixing assumption, the invariant-law map of a controlled chain is itself Lipschitz, so any closure produced from a generator is admissible in the prescribed-closure theory. A prescribed needs no such provenance to satisfy the hypotheses of Chapters 2—5 (it must only be Lipschitz and simplex-valued), yet the bridge theorem guarantees that the two routes coincide whenever the chain side has a uniformly mixing realization.

The protocol in summary. Given a domain, execute Steps 1—6 to produce the data tuple . The Chapter 2 template then gives a specific ODE on , and the theorems of Chapters 3—7 apply. To verify the instantiation, check that the general formulas recover recognizable domain objects when evaluated on the chosen data.

Let us execute the protocol once more on the Section 2.8 routing chain, to confirm that the procedure recovers the known model.

Step 1. Three states: , representing the retail hub, health discovery, and vet booking.

Step 2. Two actions: , representing retail-safe and vet-push presentation.

Step 3. Scalar actor () with state-independent features: , for all . A positive favors vet-push at every state.

Step 4. Scalar critic () with matching features: , for all .

Step 5. Rewards: retail-safe earns more at the hub (state 1), vet-push earns more in the funnel (states 2, 3). The exact values appear in Section 2.8.

Step 6. Two action-conditioned generators and encoding the customer flow: under retail-safe, customers tend to return to the hub; under vet-push, customers advance toward vet booking. The softmax mixture is a convex combination weighted by and .

The resulting system is the Section 2.8 ODE, and the Chapter 6—7 analysis applies to it.

The six protocol steps are not equally consequential. Steps 1 and 2 (states and actions) determine the dimension of the phase space. Step 5 (rewards) determines the scale and direction of the gradient. Step 6 (generators) determines the chain dynamics. Step 3, the choice of actor features, has a structural effect that the other steps do not: it determines which symmetries the coupled system has and therefore which patterns of equilibria the attractor can exhibit.

We illustrate this with three case studies drawn from the two existing models and from the recommendation model that Chapter 10 will develop.

Case 1: Antisymmetric features and distributional decoupling.

The Chapter 0 features satisfy , and the rewards satisfy , . Substituting these into Definition 2.5 and using and , the raw actor drift collapses to

The right-hand side has no . The occupancy weights sum the two state contributions; under the antisymmetric structure ( across actions and across states, with matching rewards and critic features), the score-weighted reward, critic, and entropy terms at state 1 contribute exactly the same amount as their state-2 counterparts: each state contributes per unit of its mass, so the coefficients combine into the prefactor .

The consequence for attractor geometry is sharp but not literally Cartesian: the actor-critic subsystem evolves autonomously, and the distribution is driven by the complete -history without feeding back. The full invariant object is therefore a skew-product lift over complete bounded actor-critic trajectories. The compatible -state above a base point is selected by the past trajectory that reaches that point, not by the instantaneous value alone unless an additional fiber-uniqueness argument is proved. This is why the Chapter 0 phase portrait could be drawn in the -plane alone for actor-critic stability questions: the distribution coordinate was redundant for the base dynamics.

The decoupling is an artifact of the feature structure rather than a general property. The simplest example carries a simplification that does not survive in general models: any model with features that break the antisymmetry will couple into the actor drift.

Case 2: State-independent features and constant critic coefficients.

The Section 2.8 routing features satisfy and for all states . This structure does not decouple from the actor drift; the occupancy weights still enter the gradient. It does simplify one side of the critic regression. The Section 2.8 critic features are state-independent with for every pair, so the covariance term in sums the occupancy weights to one:

The curvature coefficient is constant: it carries no or dependence at all. The simplification stops there. The forcing term retains the full per-state breakdown of the population, because the rewards differ across states; Section 2.8 displays the coupling explicitly as

The general version of this observation needs care. With state-independent critic features, the matrix depends on only through the per-action weights , and those weights still see the per-state breakdown of whenever the policy is state-dependent. The breakdown leaves the weights only when the actor features are state-independent as well, so that is the same at every state; and it leaves entirely, as in Section 2.8, when the squared critic features are equal across all pairs.

The consequence for attractor geometry is that the critic equilibrium is an explicit function, affine in for each fixed , which makes the equilibrium computation tractable.

Case 3: Block-diagonal features and independent policy axes.

Suppose the actor has dimension and the features have a block-diagonal structure: at states in one cluster, only the first component affects the policy; at states in another cluster, only the second component affects the policy. Concretely, the features at cluster-A states have the form , and the features at cluster-B states have the form .

Under this structure, the actor drift decomposes partially: the component depends on the occupancy at cluster-A states and on , while depends on cluster-B states and . The two policy axes are coupled through the distribution equation (changing shifts the population, which changes the mass available to cluster B) but not through the direct gradient coupling, provided the critic features carry the same block structure, as Chapter 10 arranges with its two-dimensional block-diagonal critic. A shared unstructured critic would couple the axes a second way: reads the critic estimate , which aggregates reward information from both clusters.

The consequence for attractor geometry is that the system can have independent attractor branches along each policy axis. Each branch can reach its own boundary equilibrium independently of the other, which is the structural prerequisite for multi-equilibrium phenomena with domain-specific meaning.

Chapter 10 uses this feature structure for its recommendation model; the domain interpretation of the two clusters and the two policy axes is developed there.

The general lesson. Feature design is a structural decision that shapes the attractor. The features encode the modeler’s hypothesis about which state-action distinctions matter, and that hypothesis determines which attractor phenomena the system can exhibit. Antisymmetric features hide the distributional coupling. State-independent features simplify the critic. Block-diagonal features create independent policy axes and enable branch-specific attractors. A modeler who wants to see multi-equilibrium phenomena should choose features that allow distinct attractor branches to form.

Step 6 of the protocol asks for a generator family . The generator depends on because the agent’s policy reshapes the environment’s transition rates: each action induces a different rate matrix, and the policy mixes those matrices state by state. In many applications, the direct way to build this family is from a domain topology: a graph whose nodes are the states and whose edges represent possible transitions.

The construction has three layers.

Layer 1: The domain graph. Draw a directed graph where an edge means that a transition from state to state is physically possible. Edges that are not in the graph correspond to transitions that cannot happen regardless of the agent’s action.

Layer 2: Action-conditioned base rates. For each action and each edge , assign a base transition rate . For non-edges, set . Complete the diagonal: . The result is a generator matrix for each action.

The base rates encode what each action does to the environment. In the Section 2.8 routing model, the retail-safe action has high return rates to the hub, while the vet-push action has high forward rates into the funnel. The domain graph is the same for both actions; the topology does not change, but the rates on each edge do.

Layer 3: The softmax mixture. The effective generator under policy is the state-dependent mixture

Because for all finite (the softmax is strictly positive) and each , the mixture inherits the generator property: off-diagonal entries are nonnegative, and the row-sum-zero condition is preserved by the convex combination.

The Lipschitz regularity of in follows from the smoothness of the softmax: is infinitely differentiable in , so is smooth and Lipschitz on the compact actor box .

Reference-state structure. The bridge theorem of Chapter 6 runs on the uniform exponential-mixing hypothesis (Assumption 6.2), and the tracking estimate of Chapter 7 runs on the pathwise contraction estimate (Assumption 7.2). Neither hypothesis mentions the domain graph. What makes them checkable from the graph is reference-state minorization (Propositions 6.6 and 7.6), and that criterion is a one-step condition: there must be a reference state and a constant such that every other state has a direct jump rate into it,

Multi-step reachability is not sufficient for this criterion. A path from to through intermediate nodes, however well its edge rates are bounded below, does not satisfy the hypothesis of Proposition 6.6, which asks for the direct edge. When some node has no direct edge to the reference state, the one-step criterion fails literally, and a multi-step extension must be proved separately: a fixed-time propagator argument whose minorization constant is built from products of rates along the connecting path. Chapter 10 carries exactly such a two-step extension for its Y-shaped content graph (Section 10.4, “Reference-state structure”), and Chapter 11 instantiates the same mechanism for its five-state hub-and-spoke network (Lemma 11.1).

In the Section 2.8 routing model, no extension is needed. The retail hub (state 1) has a direct edge from each of the other two states, and the return rates and are bounded below uniformly in , because both actions send customers back to the hub at positive rates. The one-step criterion holds with and , so Proposition 6.6 applies as stated and the bridge theorem follows.

The theorems of Chapters 2—7 produce mathematical objects: a semiflow, an absorbing set, a global attractor, equilibria, and (when the chain closure applies) a reduced system on the invariant-law graph. Each of these objects has a domain-level reading that is not visible from the abstract statement alone. This section provides a translation table.

Equilibria. An equilibrium of system (L1) is a point where the vector field vanishes. There are two ways this can happen.

An interior equilibrium occurs where the raw actor drift : the score-weighted advantage-like signal averages to zero across state-action pairs. In domain terms, this is an operating point where the policy, the value estimate, and the population distribution are mutually consistent. The agent has no first-order incentive to change its policy: the gradient signal vanishes given the current value estimate and population, so the policy is stationary there, whether or not it is optimal.

A boundary equilibrium occurs where the damping factor : the actor parameter has reached the wall of the box , and the damping suppresses the drift regardless of the advantage-like signal. In domain terms, this is an extreme policy, as specialized in one direction as the parameter box allows (about 88% on the matching action in the Chapter 0 data), that the parameter cannot move past. The raw drift at the boundary may be nonzero, and its sign tells us what the wall is doing: an outward drift means the gradient would push the policy further if the box allowed it, while an inward drift means the undamped gradient would retreat toward the interior. In the Chapter 0 data the drift at points inward (Section 0.13 computes there, because the entropy penalty dominates the reward-plus-critic signal at the wall). Either way, boundary equilibria are artifacts of the confinement mechanism, not of the incentive landscape; Exercise 9.7(d) revisits the computation.

The distinction matters for interpretation. An interior equilibrium is a genuine fixed point of the incentive structure. A boundary equilibrium is a saturation point where the system has hit a parameter limit. Both are in the attractor, but they represent different phenomena.

Multiple equilibria. When the attractor contains more than one equilibrium, the system has competing steady states. In domain terms, the algorithm’s long-run outcome can then depend on the initialization. The Chapter 0 model has three equilibria: two boundary (, policies as specialized as the parameter box allows) and one interior (, a balanced policy). There, the Section 0.14 numerics indicate that the interior equilibrium dominates: trajectories in the absorbing set bend toward it, and the boundary equilibria attract only the invariant faces of the box. Chapters 10 and 11 build models whose reward-design levers move the system toward, and numerically across, the threshold where boundary lock-in becomes attracting, so that the final behavior genuinely depends on which basin of attraction (the set of initial conditions converging to a given equilibrium, taken up in the next entry) the initial condition falls in.

For a single-equilibrium analysis, convergence is the whole story. For a multi-equilibrium system, the attractor framework buys something that convergence-rate analysis does not: a description of all possible outcomes and the geometry that separates them.

Basin boundaries. The basin of attraction of an equilibrium is the set of initial conditions that converge to it. The boundary between basins is a separating surface in the phase space. In domain terms, this is the threshold between outcomes: on one side, the system converges to one steady state; on the other, to a different one. Small changes in the initial condition near the basin boundary produce large changes in the long-run outcome.

The absorbing set. The Chapter 4 absorbing set is a compact region that all trajectories enter in finite time. In domain terms, it is the operating envelope: regardless of initial conditions, the system’s state eventually lies in . The actor stays in the box, the critic stays bounded (by coercivity), and the distribution stays on the simplex. The absorbing set gives quantitative bounds on the operating range.

Upper semicontinuity. The Chapter 7 upper-semicontinuity corollary (Corollary 7.5, built on the tracking theorem) says that when the environment responds quickly (small ), the attractor of the exact system lies inside any prescribed neighborhood of the lifted attractor of the reduced system, where the distribution instantaneously tracks the frozen invariant law. The guarantee is one-sided: the exact system cannot sustain long-run behaviors absent from the reduced system, though the reduced attractor may contain points that no exact trajectory approaches. In domain terms: if users respond to content changes much faster than the platform updates its policy, the frozen-audience analysis cannot miss any long-run outcome of the true system. The reduced system is lower-dimensional and often more tractable, so the fast-slow separation is a practical simplification.

The Model Specification Protocol is a general procedure, and the feature-attractor correspondence is a general observation. Chapters 10 and 11 execute the protocol on two domains that are large enough to exhibit attractor phenomena with genuine domain meaning.

Chapter 10: Recommendation systems and algorithmic curation. A content platform curates recommendations for users who browse across multiple content clusters. The actor is two-dimensional with block-diagonal features, in the sense developed in Section 9.3 Case 3.

The question Chapter 10 asks: when do filter-bubble configurations appear as boundary equilibria, and what extra stability evidence would be needed before calling them competing attractors of the coupled recommendation system? The analysis turns on the block-diagonal feature structure introduced in Section 9.3 and on the resulting equilibrium picture in the enlarged phase space; the state graph, the action labels, and the numerical basin protocol are developed in Chapter 10.

Chapter 11: Network routing under endogenous traffic. A routing agent directs traffic across a hub-and-spoke network with five nodes. The state space extends the Section 2.8 three-state chain to include secondary destinations reachable through the primary nodes. The action space has two routing modes: left-bias and right-bias. The actor is scalar with state-independent features, as in Section 2.8.

The question Chapter 11 answers: when a routing policy shapes the traffic distribution, which traffic configurations are self-reinforcing? The answer is a regime story: the balanced interior operating point and the lock-in configurations at the boundary compete, the reward-design levers control which of them attracts, and the fast-slow reduction shows that when traffic equilibrates quickly, the reduced actor-critic system governs the long-run routing pattern.

Both chapters follow the same protocol, use the same theorems, and produce qualitatively different attractor pictures. The difference comes entirely from the domain data: the state graph, the features, and the rewards.

The Model Specification Protocol of Section 9.2 takes a domain question through six steps and returns a data tuple that fully determines the Chapter 2 ODE on , with traded for in the controlled-chain presentation. The feature-attractor case studies of Section 9.3 show how Step 3’s symmetry choices control which equilibrium patterns the attractor can carry, and the translation table of Section 9.5 reads each abstract object (equilibrium, basin, absorbing set, upper semicontinuity) back into domain language. The chapters that follow exercise the protocol on two domains; the theorems of Chapters 3—7 apply to each instantiation without modification.

Exercise 9.1 (Verify: protocol execution on a new domain). Consider a three-state patient-triage model with state set (state 1 = healthy, state 2 = moderate, state 3 = severe) and action set (conservative treatment, aggressive treatment).

(a) Execute Steps 1—5 of the Model Specification Protocol. Choose scalar actor and critic (), assign features , for all (state-independent), and design a reward table where aggressive treatment is rewarding in the severe state but has a side-effect cost in the healthy state.

(b) Construct a closure map that assigns a higher proportion of healthy patients when treatment is more aggressive (patients recover faster). Verify that takes values in and compute its Lipschitz constant.

(c) Write the complete prescribed-closure ODE (take ) and verify that it has the form of Definition 2.10.

Exercise 9.2 (Compute: antisymmetric decoupling). For the Chapter 0 model, compute the raw actor drift explicitly as a function of , , and . Show that the expression is independent of . Identify the algebraic mechanism that causes the decoupling: show that the two per-state contributions are equal, so the -weighted sum collapses through .

Exercise 9.3 (Connect: breaking antisymmetry). Modify the Chapter 0 actor features to , , , (breaking the cross-state antisymmetry). Compute and verify that it now depends on . Describe qualitatively how the phase portrait changes: does the system still decouple?

Exercise 9.4 (Break: removing state-independence). In the Section 2.8 routing model, change the critic features at state 1 (the retail hub) to , , while keeping the features at states 2 and 3 unchanged.

(a) Show that the critic coefficient matrix becomes , so the per-state breakdown of no longer drops out of the curvature coefficient.

(b) Now restore the critic features and instead change the actor features at state 1 to , . Show that still: the occupancy weights sum to one and every squared critic feature equals one, so no choice of actor features can move . Which parts of the system do change?

Exercise 9.5 (Verify: generator property). For the Section 2.8 routing chain, write the generator explicitly as a matrix whose entries depend on through the softmax. Verify the generator property: all off-diagonal entries are nonnegative and each row sums to zero. Compute the Lipschitz constant of on .

Exercise 9.6 (Extend: closure map for the triage model). In the triage model of Exercise 9.1, the closure map specifies the target patient distribution as a function of the treatment policy. Suppose the aggressive treatment cures moderate patients with probability proportional to (the logistic function of ). With states ordered as , let

so the first component is the healthy mass and the moderate and severe masses split the remainder evenly. Verify that for all , compute the Lipschitz constant, and write the resulting distribution equation with relaxation rate .

Exercise 9.7 (Connect: domain interpretation of boundary equilibria). In the Chapter 0 model, the boundary equilibrium at has the platform as committed to the matching strategy as the parameter box allows. Using the translation table of Section 9.5, answer:

(a) What policy does represent? Compute for each state-action pair.

(b) What audience distribution does this equilibrium induce? Compute .

(c) What is the critic’s value estimate at this equilibrium? Compute at .

(d) Is the raw actor drift zero at this equilibrium, or is the equilibrium created entirely by the damping ?