Chapter 6: Bridge To The Genuine Controlled-Chain Closure

Chapter 5 completed the prescribed-closure attractor theorem. Starting from the closure map , we obtained a well-posed autonomous system on the enlarged phase space , built a compact absorbing set, and constructed a unique compact global attractor. The mathematics was complete, but one modeling ingredient was still abstract: the closure map was assumed Lipschitz and simplex-valued, not derived from the controlled Markov chain.

This chapter explains where that map comes from. In the routing example from Chapter 2, this means the stationary customer mix under a frozen routing policy is not something we invent–it is determined by the transition structure of the platform.

For each frozen actor parameter , the generator from Chapter 2 defines a finite-state continuous-time Markov chain. If that frozen chain mixes uniformly fast, then it has a unique invariant law , and the map is Lipschitz. That map is the genuine controlled-chain closure. Once it is identified, system (L1) from Chapter 2 is no longer driven by an arbitrary target law: its relaxation target can be chosen canonically as the stationary distribution of the frozen chain.

There are two conceptual points to keep separate.

First, this chapter is still about a frozen-chain object. We fix , ask for the stationary law of the corresponding chain, and study how that law varies with . Second, the resulting bridge theorem does not yet analyze the exact fast law equation . It only identifies the correct target for the relaxed prescribed-closure equation . The finite-timescale comparison between the exact chain and the reduced closure is the task of Chapter 7.

The chapter has five steps. Section 6.1 defines the frozen invariant law and computes it in the Chapter 0 chain and in the three-state retail-to-vet routing model from Chapter 2, Section 2.8. Section 6.2 introduces the uniform exponential-mixing hypothesis and shows that it gives existence, uniqueness, and exponential convergence to the invariant law. Section 6.3 proves that the map is Lipschitz. Section 6.4 states that result as the bridge theorem: the invariant-law map is an admissible closure map for the prescribed-closure attractor theorem. Section 6.5 gives a concrete reference-state minorization criterion that makes the mixing hypothesis checkable, and applies it to the routing chain.

The generator family introduced in Chapter 2 describes how the state distribution evolves when the actor parameter has value . If the actor were held fixed, the distribution would evolve by

The first question is therefore the stationary one: what probability vector, if any, remains unchanged under this frozen evolution?

Definition 6.1 (Frozen invariant law). Fix . A vector is called an invariant law of the frozen chain if

Equivalently, is a stationary solution of the frozen law equation .

Once the invariant law exists, it determines a genuine stationary occupancy on state-action space:

This is the first place in the notes where we can legitimately use the symbol without a dynamic . In Chapters 0-5, the relevant object was always the product occupancy , because the law was a genuine state variable. Here we are freezing and looking only at the stationary law of the resulting chain. Section 6.4 will make this identification rigorous by showing that the invariant-law map satisfies the prescribed-closure requirements.

For the reinforcement-learning reader. Freezing means pretending that the policy has stopped learning and asking what state distribution the environment would settle into under that fixed policy. The invariant law is therefore the long-run population mix generated by one policy, not by the full learning dynamics.

For the dynamical-systems reader. The notation is the adjoint generator acting on distributions. The equation is the finite-state analogue of a stationary Fokker-Planck equation.

The definition is abstract, so let us compute two examples.

For the Chapter 0 system, the natural two-state chain (introduced in Exercise 2.2 of Chapter 2) has transition rates

so the generator is

Write . The stationary equation becomes

Solving gives

This is exactly the closure map that appeared by hand in Chapter 0. So in that toy example, the prescribed closure was not a guess: it already was the frozen invariant-law map of the exact two-state chain.

The routing model of Section 2.8 has three states:

• core retail,
• health discovery,
• vet booking,

and two action-conditioned generators and . With the softmax policy and , the frozen generator is

For notational convenience, write

Then

If , the stationary system reads

together with .

The second and third equations give

Substituting into the normalization condition yields the explicit formula

where

All three components are strictly positive for every , so the formula indeed defines a probability vector in .

The formula is less transparent than in the two-state case, but the meaning is clear. Under a frozen routing policy, the user population settles into a stable split across retail, health discovery, and booking. The more aggressively the policy pushes the veterinary funnel, the more the stationary law shifts mass away from the retail hub and toward the deeper service states. That shift is what the bridge theorem will later feed back into the closure map.

At this stage, however, we have only computed candidate stationary laws in the examples. We have not yet proved that every frozen chain in the general theory has a unique invariant law, nor that the map is regular. Those are the next two tasks.

An invariant law is useful only if the frozen chain actually forgets its initial state and converges to it. For a single fixed , irreducibility of the finite-state chain is enough to guarantee convergence. But the bridge theorem needs more than pointwise convergence: it needs control that is uniform over the whole actor box . Otherwise the invariant-law map can change too violently with to serve as a Lipschitz closure.

The correct space for the mixing estimate is the zero-mass subspace

Differences of probability vectors live in , so contraction on is exactly the statement that two distributions become indistinguishable as the chain runs.

Assumption 6.2 (Uniform exponential mixing). There exist constants and such that

The -norm is the natural one here because on the probability simplex it is twice the total-variation distance.

This assumption says that every frozen chain contracts differences of probability distributions at the same exponential rate, up to the same prefactor, uniformly over the actor box.

Before proving the main consequence, we import one finite-state Markov fact.

Imported fact. For every generator and every , the matrix is column-stochastic: it preserves nonnegativity and total mass. In particular,

This semigroup property of finite-state continuous-time Markov chains is proved in Norris, Markov Chains, Chapter 2.

We can now prove existence, uniqueness, and exponential convergence.

Theorem 6.3 (Uniform mixing implies a unique invariant law). Assume Assumption 6.2. Then for each there exists a unique vector such that

Moreover, for every initial law ,

Since any two probability vectors satisfy , this implies the coarser but sometimes more convenient bound .

The proof proceeds in three stages: each frozen orbit is Cauchy as ; its limit is stationary; and the contraction estimate on forces uniqueness.

Proof. Fix and .

First, we show that the frozen orbit is Cauchy. For , the semigroup property gives

The vector in parentheses has total mass zero, because both and are probability vectors. Therefore it belongs to , and Assumption 6.2 together with the general bound on differences of probability vectors (which is what is available at this stage, before has been constructed) yields

The cruder prefactor is the right one here because the Cauchy estimate compares two orbit times rather than orbit and limit; the tighter form re-enters once has been identified, in the convergence step at the end of the proof.

Hence is a Cauchy family in as . Let denote its limit. Since each belongs to and is closed, we have .

Second, we prove that the limit is stationary. Fix . By the semigroup property,

where continuity of the matrix exponential justifies passing the limit through . Differentiating the identity at gives .

Third, we use the contraction estimate to force uniqueness. Let be any other invariant law: . Then

Since , Assumption 6.2 implies

Letting forces . So the invariant law is unique.

Finally, for any initial , the difference belongs to . Since is stationary,

which is the stated convergence estimate.

For the two-state chain, every vector in has the form . A direct computation gives

because . Therefore

So the Chapter 0 chain satisfies Assumption 6.2 with

This is stronger than mere convergence: the contraction is exact and uniform.

For the three-state routing chain, the eigenvalue computation is less pleasant. The chain is small enough that one could still diagonalize , but the resulting formulas are not enlightening. Section 6.5 will give a better route: a direct minorization criterion that proves the required mixing without requiring explicit spectral calculations.

The closure map in Chapter 2 was required to be Lipschitz on the actor box, and Assumption 2.8 from Chapter 2 gave the generator regularity needed for the chain side. Existence of an invariant law is therefore not enough. To use as a genuine closure map, we must know that small changes in produce proportionally small changes in the stationary law.

The key tool is a resolvent on the zero-mass subspace .

Theorem 6.4 (Lipschitz regularity of the invariant-law map). Assume Assumption 2.8 from Chapter 2 and Assumption 6.2. Then the map

is Lipschitz with respect to the -norm:

The mechanism is the same as in many perturbation arguments. We solve on the zero-mass space by integrating the semigroup, then apply that inverse to the difference between the two stationary equations.

Proof. Fix . For , define

The integral converges absolutely in , because Assumption 6.2 gives

So is a bounded linear map on satisfying

We claim that on . First note that preserves the zero-mass space: if , then , so . Thus is well-defined. Indeed, if , then

Therefore

Since , Assumption 6.2 gives as . Hence

Now fix . The stationary equations are

Subtracting them gives

Because and are probability vectors, the difference belongs to . The right-hand side also belongs to , because each vector has total mass zero. Apply to both sides:

Taking -norms and using Assumption 2.8,

because . This is the desired Lipschitz bound.

For the Chapter 0 chain,

so

Thus the exact Lipschitz constant is .

On the other hand, the generator family satisfies

so one may take . Together with and , Theorem 6.4 gives

So in this example the abstract perturbation bound is exactly sharp.

For the routing model,

so

Since the logistic function satisfies on , one has

A direct -operator norm computation gives

Therefore the routing generator family is Lipschitz with, for example,

Once uniform mixing is known, Theorem 6.4 then gives a concrete Lipschitz bound for the stationary customer-mix map .

We can now return to the question that Chapter 2 deliberately postponed. The prescribed-closure theory required a Lipschitz closure map . This chapter has now produced a canonical candidate:

the invariant law of the frozen controlled chain.

The next statement is the bridge theorem in the lecture-note setting.

Corollary 6.5 (Bridge to the genuine controlled-chain closure). Assume the standing data hypotheses from Chapter 2 (in particular Assumption 2.7), as well as Assumption 2.8 and Assumption 6.2. Define

Then is Lipschitz on and simplex-valued. Moreover, let be the coordinatewise clipping map onto the actor box,

Then the clipped extension is a globally Lipschitz map from to with the same Lipschitz constant , and it agrees with on . Consequently is an admissible closure map in the sense of Definition 2.9, with used for the ambient well-posedness argument, and the prescribed-closure system

satisfies the hypotheses of Chapters 2—5. In particular, the global attractor theorem of Chapter 5 applies to this system.

Proof. Theorem 6.3 gives for every , and Theorem 6.4 gives the Lipschitz estimate on . One step remains before Chapters 3—5 can be invoked: the well-posedness argument of Chapter 3 consumes a globally Lipschitz ambient extension of the law field (Lemma 3.4) and runs the Picard iteration in the ambient space (Proposition 3.6) before restricting to the invariant phase space. The clipping map supplies that extension. Each coordinate of applies the scalar map , which is -Lipschitz, so is -Lipschitz and fixes every point of . The composition is therefore simplex-valued, globally Lipschitz with constant , and restricts to on . With this extension, the standing data assumptions from Chapter 2 hold, and the results of Chapters 3—5 apply to the prescribed-closure system. On the phase space , where ranges over , the closure target is exactly , which is the system displayed above.

This is the point at which the prescribed closure stops being arbitrary. The relaxed distribution equation

can now be read concretely as

meaning that the current distribution is pulled toward the stationary law of the frozen chain generated by the current policy.

Two scope boundaries are worth marking.

The bridge theorem does not reduce the system to -variables alone. The actor and critic equations still depend on the current dynamic distribution , just as they did in Chapter 2; the only change is that the target of the relaxation equation is now chosen canonically.

The relaxation model itself, the equation , is unchanged; as noted at the chapter opening, the exact law equation and its finite-timescale comparison with the reduced closure are the subject of Chapter 7.

For the Chapter 0 chain, the bridge theorem recovers exactly the closure map already used there:

So the toy model was not using a guessed closure after all. It was already the relaxed system driven by the true stationary law of the frozen two-state chain.

For the retail-to-vet routing chain, the bridge theorem says that the closure map is not something we invent by hand. It is the stationary customer mix of the frozen routing process:

That formula gives the canonical target in the relaxed law equation . The resulting prescribed-closure system still evolves in three coupled coordinates, but it is now grounded in the exact stationary behavior of the policy-induced chain.

The bridge theorem depends on Assumption 6.2, the uniform exponential-mixing hypothesis. In abstract form, that assumption is exactly what we want. In practice, however, we need a structural criterion that can be checked directly from the transition rates.

It is worth keeping the downstream chain in view from the start. Once a reference-state minorization condition is verified, it implies uniform exponential mixing of the frozen chain (Assumption 6.2 holds via Proposition 6.6), which in turn makes the invariant-law map Lipschitz (Theorem 6.4) and therefore an admissible prescribed closure (Corollary 6.5); the same reference-state mechanism reappears in Chapter 7 as the pathwise contraction estimate that drives the singular-limit tracking and upper-semicontinuity results. The rest of this section verifies the first link in this chain.

The simplest such criterion for our finite-state setting is a reference-state minorization condition. The idea is that one state acts as a universal anchor: from every other state, there is always a uniformly positive direct jump rate into . If that holds, then over a short but fixed time window, every initial distribution puts a definite amount of mass into the anchor state. Repeating the argument block by block produces exponential contraction. If , then , so every difference of probability vectors is the zero vector and the mixing inequality holds vacuously; the interesting case is .

Proposition 6.6 (Reference-state minorization implies uniform mixing). Assume there exist a distinguished state and a constant such that

Define

Then for every , each column of the frozen propagator

dominates . Consequently Assumption 6.2 holds: if , it holds with

and if , then and the mixing inequality is trivial (for instance with and ).

The proof has three stages. First, we use Gronwall’s inequality on the diagonal components to show that the mass at any initial state decays no faster than . Second, we feed that lower bound into an integrating-factor argument to show that after one time block of length , every initial state sends at least mass into the reference state . Third, we decompose the propagator as a column-stochastic contraction plus a rank-one remainder that annihilates , turning the one-block mass bound into exponential contraction on the zero-mass subspace.

Proof. Fix . For each state , let solve

Thus is the -th column of .

We first bound the -component at time .

If , then the only way mass can leave is through the total exit rate from , which is bounded by . Therefore

Gronwall’s inequality gives

Now suppose . The -component satisfies

so again by Gronwall,

The -component receives inflow from state at rate at least , while all other terms can only help. Therefore

Multiply by the integrating factor :

Integrating from to and using gives

Hence

So every column of dominates .

If , then and the mixing inequality holds with any constants, so assume from now on that . Choosing any state , the minorization hypothesis gives

so , and the divisions by below are legitimate.

Define

The column lower bound shows that has nonnegative entries, and both and are column-stochastic, so is also column-stochastic. If , then , hence

Since every column-stochastic matrix is nonexpansive in ,

Therefore

Now fix , and write

Because is column-stochastic, it is nonexpansive on . Iterating the block contraction on the first full blocks gives

Finally,

So Assumption 6.2 holds with

This proves the claim.

For the Chapter 0 chain, choose the reference state . Then

So one may take

The maximal total exit rate is

Hence

These constants are much worse than the exact spectral-gap values , , but that is not the point. The minorization criterion is structural, not sharp. It proves mixing from the presence of a uniformly reachable reference state, without diagonalizing the generator.

Now return to the three-state routing example from Section 2.8. We choose the retail hub as the reference state. The direct return rates into the hub are

Since for all , we have the uniform lower bounds

So we may take

The total exit rates from the three states are

Therefore

because the middle branch dominates the other two for every (its gaps to them are and , both positive since ) and is increasing. The proof of Proposition 6.6 uses only as an upper bound on the total exit rates, so any larger value yields valid, slightly weaker constants. We may therefore take and work with rounder numbers.

With , the minorization constants are

The important point is not the numerical value of these constants. The important point is the mechanism they encode. No matter how strongly the actor pushes the veterinary funnel, both the discovery state and the booking state retain a uniformly positive direct route back to the retail hub. That single structural fact is enough to force uniform mixing of the frozen chain.

This is exactly the kind of theorem-friendly interpretation we wanted from the business example. A loose slogan for this structure would be that the core business cannot disappear. That slogan is too loose to be mathematics. The mathematically correct statement is sharper: the retail hub is a uniformly reachable reference state, so the frozen controlled chain mixes uniformly exponentially, which in turn makes the stationary customer-mix map well-defined and Lipschitz.

The same geometry will return in Chapter 7. There the actor is no longer frozen and the fast law equation is time-dependent through , but the same reference-state mechanism will drive the pathwise contraction estimate.

This chapter supplied the missing interpretation of the closure map.

For each frozen actor parameter , the generator defines a finite-state continuous-time Markov chain. Under uniform exponential mixing, that chain has a unique invariant law , and the map is Lipschitz. That map is the genuine controlled-chain closure . System (L1) from Chapter 2 may therefore be read concretely as a relaxed actor-critic-law system in which the dynamic distribution is pulled toward the stationary law of the frozen chain.

The examples show both ends of the spectrum. In Chapter 0, the invariant-law map is so simple that it can be written down immediately: . In the three-state retail-to-vet routing model, the formula is more elaborate, but the same attractor theorem still applies. The stationary customer mix is explicit, the generator family is Lipschitz, and the return-to-hub structure gives a direct minorization criterion.

What this chapter has not done is analyze the exact system with finite timescale separation,

In that system, is not replaced by : the law variable remains dynamic and may lag behind the stationary law by an amount that depends on . Chapter 7 is about that lag: finite-time tracking of the exact flow by the reduced closure, upper semicontinuity of attractors, and the upgrade of the minorization argument from frozen chains to non-autonomous fast equations.

Exercise 6.1 (Chapter 0 invariant law). Starting from the two-state generator

solve and directly. Verify that the result equals the closure map used in Chapter 0.

Exercise 6.2 (Exact mixing rate in the two-state chain). Show that every has the form , and verify directly that . Conclude that the exact Chapter 0 mixing constants are and .

Exercise 6.3 (Exact Lipschitz constant). Use the explicit Chapter 0 formula for to compute the exact Lipschitz constant of in the -norm. Then compute for the two-state generator family and verify that Theorem 6.4 is sharp in this example.

Exercise 6.4 (Routing-chain invariant law). For the Section 2.8 routing generator, derive the formula

from the stationary system . For each of the three stationary equations, identify the inflow and outflow terms and interpret them in terms of user transitions between retail, discovery, and booking.

Exercise 6.5 (Minorization constants for the routing chain). Verify the lower bounds and , compute the exact supremum defining , and explain why the rounded upper bound may be used in its place. Then recover the constants from Proposition 6.6.

Exercise 6.6 (When minorization fails). Construct a finite-state chain family in which some state can become nearly absorbing as varies, so that no uniform reference-state lower bound is possible. Explain why the proof of Proposition 6.6 breaks down. (Hint: identify the step in the proof where the uniform lower bound on the jump rate into the reference state is used.)

Exercise 6.7 (Two occupancies). In one paragraph, explain the difference between and . At what point in the notes is it legitimate to replace the first by the second, and why is that replacement still not automatic in the exact finite- system?

Exercise 6.8 (Loss of uniform mixing). On , consider the two-state chain family with transition rates , so both rates vanish at . Using the identity on from the two-state computation in Section 6.2 (see also Exercise 6.2), show that no constants and satisfy Assumption 6.2 for this family, and describe the set of invariant laws at . Next, keep but set ; at state is absorbing, so the chain is reducible there. Show that Assumption 6.2 nevertheless holds for this second family, with and . Conclude that reducibility at a single parameter value does not by itself destroy uniform mixing in continuous time: the contraction rate on is the total jump rate , and only a vanishing total rate can break it. Finally, identify where the uniform constants are consumed downstream–the resolvent bound in the proof of Theorem 6.4 and the Lipschitz constant in Corollary 6.5–and check that for the first family the invariant-law map is in fact constant on , so the conclusion of Theorem 6.4 survives by symmetry even though its proof breaks down. Uniform mixing is a sufficient condition, not a necessary one.