Chapter 7: Fast-Slow Reduction

Chapter 6 supplied the missing interpretation of the closure map. Under a uniform exponential-mixing hypothesis for the frozen generators , the stationary law exists uniquely for each frozen actor parameter , depends Lipschitzly on , and therefore defines the genuine controlled-chain closure . That closed the bridge from the prescribed-closure system to the actual frozen Markov chain.

In the routing example, the remaining question is immediate: a platform may adjust its policy slowly enough that the customer mix has time to re-equilibrate between noticeable policy changes. In that regime, the population law is the fast variable and the actor-critic pair is the slow variable. But a gap remains between the bridge theorem and the exact dynamics.

The exact enlarged-state dynamics do not replace the law variable by . They evolve the law by the finite-timescale equation

so the distribution remains a genuine state coordinate that may lag behind the instantaneous stationary law. The informal two-timescale slogan says that when is small, the law should relax quickly and the full trajectory should stay close to the invariant-law reduction. The point of this chapter is to turn that slogan into a theorem.

This chapter proves three things.

1. We write the exact fast-law system and the reduced invariant-law system in a common notation and identify the lifted reduced attractor.
2. Under a pathwise contraction hypothesis for the non-autonomous fast law equation, we prove a finite-time tracking estimate and then deduce upper semicontinuity of attractors as .
3. We show that the reference-state minorization mechanism from Chapter 6 upgrades to the required pathwise contraction estimate, so the abstract hypothesis can be checked directly in the routing model.

Two scope boundaries matter from the start.

First, the reduction proved in these notes is a one-fast / one-slow theorem: the actor and critic evolve on the scale, while the law evolves on the scale. In the stochastic-approximation literature one often introduces a second small parameter for a separate critic scale. That is a natural next question, but it is not part of the theorem we prove here.

Second, this chapter completes the stationary autonomous story without altering the deterministic core that the preceding chapters have assembled. The extension directions sit in Chapter 8.

The chapter proceeds in four steps. Section 7.1 sets up the exact and reduced systems. Section 7.2 proves the finite-time tracking theorem. Section 7.3 turns finite-time tracking into upper semicontinuity of attractors. Section 7.4 shows how reference-state minorization implies the pathwise contraction hypothesis.

In the routing regime described above, the population law is the fast variable and the actor-critic pair is the slow variable. The chapter’s first task is to write those two layers side by side.

As fixed in the chapter opening, no second small parameter enters: the actor and critic evolve on the timescale, while the law evolves on the timescale. The reduced system will therefore be the system obtained by inserting the frozen invariant law into the actor and critic drifts.

Assume from now on the hypotheses of Chapter 6, so that the map is well-defined and Lipschitz on . Write

Definition 7.1 (Exact and reduced systems). For , the exact fast-law system on is

The corresponding reduced invariant-law system on is

If and , we write

for the exact trajectory and

for the reduced trajectory. The lifted reduced trajectory through is

The reduced system is not a new model invented for Chapter 7. Its drifts are the Chapter 2 actor and critic drifts, evaluated at the genuine invariant-law closure that Chapter 6 constructed, and the law has been eliminated as a state coordinate rather than retained as a relaxing one. The reduced system is therefore not system (L1) with closure : that system keeps a dynamic relaxing toward as a third coordinate, and the graph is not invariant for it, since on the graph the relaxation drift vanishes while keeps moving. Both the exact system and the reduced one consequently need well-posedness and dissipativity checks of their own. The checks are short, because every Chapter 3—5 mechanism depends only on structure that both systems share. Two compact sets organize them; with the critic absorbing radius from Chapter 4, write

Theorem (Exact-system well-posedness and attractor existence). Assume Assumptions 2.7 and 2.8, including the generator regularity assumptions on . For every , the exact fast-law system of Definition 7.1 generates a continuous semiflow on The compact set is forward invariant and absorbs every bounded subset of , with an absorption time independent of . Consequently each exact semiflow has a unique compact global attractor

Proof. The exact system differs from system (L1) in exactly one place: the law equation is the generator equation rather than the relaxation equation . We verify the three Chapter 3—5 facts in turn.

Local well-posedness. Assumption 2.8 gives the Lipschitz bound for only on the box , while the Picard-Lindelöf argument of Chapter 3 (Proposition 3.6) needs a locally Lipschitz field on the ambient Euclidean space. Let be the coordinatewise clipping map and extend the generator family by . Because is -Lipschitz, the map is Lipschitz on all of , so the law component , which is Lipschitz in uniformly over bounded and linear in , is locally Lipschitz. The actor and critic components are the ones from Chapter 3 (Lemmas 3.2 and 3.3), so the full ambient field is locally Lipschitz, and the Picard-Lindelöf step of Proposition 3.6 applies verbatim: every initial datum in generates a unique maximal solution.

Actor box and simplex. The actor equation is unchanged, and the barrier proof of Proposition 4.1 uses only the damping factorization with a continuous input ; it applies verbatim, so remains forward invariant. For the law equation, the generator structure replaces the variation-of-constants argument of Proposition 4.2. Total mass is conserved because gives

For nonnegativity, let , which is finite because is Lipschitz on the compact box (Assumption 2.8). Set , so that

with a coefficient matrix whose entries are all nonnegative: the off-diagonal entries of are the off-diagonal rates, and the diagonal entries are nonnegative by the choice of . The Picard iterates

therefore map nonnegative functions to nonnegative functions and converge uniformly on compact intervals to the unique solution (the Picard argument of Proposition 3.6, with measurable time dependence and a field linear in ), so componentwise. Hence remains forward invariant. This argument uses only that takes values in , so it applies along arbitrary measurable actor paths; Section 7.4 reuses it in exactly that form.

Critic, semiflow, absorption. The energy proof of Proposition 4.4 uses only the coercivity bound and the forcing bound , both valid because stays in . The same differential inequality therefore gives , no coordinate blows up, and the solution operators assemble into a continuous semiflow on for every , exactly as in Proposition 4.6. Since , the set is forward invariant, and the absorption-time estimate of Proposition 4.5 depends only on the initial critic norm, so absorbs bounded sets at a time independent of . Because the omega-limit construction of Chapter 5 uses only a continuous semiflow together with a compact forward invariant absorbing set, it applies to unchanged: each exact semiflow possesses a unique compact global attractor .

The reduced semiflow and its attractor. The same three mechanisms run on . By Theorem 6.4, the map is Lipschitz on ; composing with the clipping map extends it to a Lipschitz map on , and inserting that map into the locally Lipschitz drifts of Lemmas 3.2 and 3.3 keeps the reduced field locally Lipschitz on . The Picard-Lindelöf step of Proposition 3.6 therefore gives unique local solutions. The reduced actor equation retains the damping factorization , so Proposition 4.1 applies verbatim and is forward invariant. Because for every , the forcing and coercivity bounds and hold, so the energy estimate of Proposition 4.4 applies and yields global existence, the continuous reduced semiflow , and the absorbing radius . Hence is compact, forward invariant, and absorbing for on , and the Chapter 5 construction gives the compact global attractor . Its lift

is the asymptotic target with which the exact attractors will be compared. It is equivalently the global attractor of the lifted reduced semiflow on the invariant-law graph.

For the reinforcement-learning reader. The reduced system is the rigorous version of the familiar sentence “the chain equilibrates between actor updates.” It does not say that the exact distribution is always stationary. It says that we have a well-defined slow system living on the invariant-law graph, and we can now ask for a theorem comparing the exact dynamics with that graph.

For the dynamical-systems reader. The reduced system is a vector field on the lower-dimensional phase space , while the exact system lives on the larger phase space . The comparison object is therefore the lift of a -trajectory to the invariant graph . Upper semicontinuity later in the chapter is a robustness statement for attractors under this singular perturbation.

The hand-computable two-state chain from Chapters 0 and 6 is the cleanest place to see the setup. Its frozen invariant law is

so if we write only the first component , the exact law equation is

In the asymmetric Chapter 0 model of Section 0.15 (Breaking the Symmetry), the slow equations are

The reduced system is obtained by replacing with its frozen equilibrium value :

where and . The point is visible without any theorem: the dynamic law coordinate has disappeared from the slow equations, but its influence remains through the stationary substitution .

In the retail-to-vet routing model of Chapter 2, Section 2.8, the exact law equation is

with

Chapter 6 computed the frozen invariant law explicitly:

where , , , are the rate combinations and is the normalizer introduced with the routing generator in Chapter 6. This is a transition rate of the chain, not the critic forcing of Definition 7.1.

The reduced routing system is therefore the Section 2.8 actor-critic system with the current customer mix replaced by this stationary customer mix . In business language, the exact model keeps track of the currently observed browsing population, while the reduced model replaces that population by the one a frozen policy would produce after the transient has washed out.

The reduction theorem ahead composes three separate mechanisms: the reduced flow and its lifted phase-space geometry, the non-autonomous law evolution, and the comparison estimates that turn one into the other.

The exact and reduced systems are now on the table. The real question is whether trajectories of the exact system stay close to the lifted reduced system on finite time intervals once the fast law has had time to relax. That is the content of the tracking theorem.

Chapter 6 gave exponential convergence for each frozen chain. That is not yet enough for the exact reduction theorem, because in the exact system the actor parameter moves. The fast law therefore solves a non-autonomous equation driven by the whole path , not by a single frozen value of .

The right hypothesis is a pathwise version of the Chapter 6 mixing estimate. It asks for exponential decay on the zero-mass space along every measurable actor path.

Assumption 7.2 (Pathwise fast-state contraction). In addition to Assumption 6.2, there exist constants and such that for every measurable path , the evolution family

of the non-autonomous fast equation

satisfies

This assumption is exactly the extra ingredient that Chapter 6 did not need. Assumption 6.2 supplies the frozen invariant law and its Lipschitz dependence through Theorem 6.4. Assumption 7.2 is different: it says that the fast law remains contractive along a moving actor path, so the law stays stable even while the actor drifts.

Section 7.4 shows that this pathwise hypothesis follows from a uniform reference-state minorization: a fixed anchor state is reached with positive probability on every block of length , no matter how the actor drifts during the block. The chain minorization \Rightarrow$ mixing \Rightarrow$ Lipschitz closure \Rightarrow$ tracking is therefore the through-line from the structural hypothesis on the generator to the comparison estimate of Theorem 7.3.

We can now state the main comparison theorem.

Theorem 7.3 (Finite-time tracking of the reduced flow). Assume the standing finite-state hypotheses of Chapter 2, Assumptions 2.7 and 2.8, Assumption 6.2, and Assumption 7.2. Let be the Lipschitz invariant-law map supplied by Theorem 6.4. Fix . Then there exists a constant , independent of , such that the following holds.

For , write

let

and let

be the lifted reduced trajectory. Then for every ,

and

Consequently, for every fixed ,

The proof has two steps. We first compare the exact law with the moving target . That is the genuine fast-slow part of the argument: the non-autonomous fast equation contracts, while the target graph moves only at speed. Second, once the law defect is controlled, the actor and critic equations are a Lipschitz perturbation problem on a compact set, so Gronwall’s inequality converts the law estimate into a full phase-space estimate.

Proof. Because is forward invariant for the exact flow (the exact-system theorem of Section 7.1), the exact trajectory starting at stays in , and the continuous function is bounded along it. Set

Then

By Theorem 6.4, the invariant-law map is Lipschitz on . Let be a Lipschitz constant such that

Define the moving reference law

Since is absolutely continuous and is Lipschitz, the map is absolutely continuous as well, and for almost every ,

Now set

Both and are probability vectors, so . Moreover,

for almost every . This is the place where Assumption 7.2 enters. Two features of the equation matter for the integral representation. First, the forcing also lies in , because is a probability vector for every ; together with the fact that the evolution family preserves , this keeps the integrand below in along the whole interval. Second, the equation reads , or equivalently , so the homogeneous part is the rescaled fast equation whose evolution family is exactly , while the inhomogeneous forcing is without an extra factor of . Writing , variation of constants applied to this rescaled equation gives

The factor of on the left-hand side of the original equation has been absorbed into the contraction rate of ; it does not appear as an additional prefactor on the forcing term.

Taking -norms and using Assumption 7.2,

The derivative bound for yields

Because

we obtain

This is the first displayed estimate after enlarging the constant and renaming it .

We now compare the slow variables. Define

The exact and reduced trajectories remain in the compact sets and , both forward invariant by Section 7.1, so the maps , , and are uniformly Lipschitz there. Choose constants such that for all admissible arguments,

Since and on the absorbing set, the critic equation gives

for some constant . Similarly, the actor equation yields

for some constant . Adding the two inequalities and using

we obtain a constant such that

for every .

Gronwall’s inequality now gives

Insert the law estimate already proved:

Since both and lie in , their -distance is at most . Enlarging the constant again gives the clean bound

To compare full phase points, use the product metric from Chapter 2:

Combining the -bound with the law estimate gives the second displayed estimate.

Finally, if is fixed, then the initial-layer factor converges to zero uniformly for as . Because the remaining term is , the stated uniform convergence on follows.

The theorem says exactly what the informal timescale argument had promised, but with the right caveat attached. There is an initial layer of width during which the law relaxes to the invariant-law graph. After that layer, the full trajectory remains close to the lifted reduced trajectory on any fixed finite time window.

For the two-state chain, the pathwise hypothesis is especially transparent. Any has the form , and for every measurable path ,

So the non-autonomous equation on is actually autonomous:

which yields the exact formula

Thus Assumption 7.2 holds with the sharp constants

In the scalar law coordinate, the tracking estimate becomes

This is a fully visible initial-layer estimate. The defect from the invariant-law graph decays exponentially fast on the -scale, and the remaining mismatch is proportional to the slowness of the actor motion.

In the routing model, the theorem says that once the customer-mix dynamics are fast enough, the exact browsing population stays close to the stationary customer mix generated by the current policy. That is the rigorous content of the sentence “the customer mix equilibrates between policy updates.” The theorem is stronger than a local heuristic because it controls the full phase-space distance between the exact trajectory and the lifted reduced trajectory on every fixed finite horizon.

The finite-time comparison theorem is the singular-perturbation core of the chapter. The next section turns that finite-horizon estimate into a statement about the asymptotic objects themselves.

Theorem 7.3 compares trajectories on a fixed time interval. Attractors concern behavior as . To pass from one to the other, we need one more ingredient: the uniform absorbing set , which Section 7.1 verified for the exact system at every . Once both the exact and reduced systems are trapped in fixed compact regions, finite-time closeness on a suitably chosen horizon becomes enough to compare their attractors.

This is the point where the language of robustness enters. From the dynamical-systems side, upper semicontinuity means that a small perturbation of the vector field cannot produce pieces of asymptotic behavior far away from the unperturbed attractor. From the reinforcement-learning side, it means that if the state distribution mixes quickly, then the long-run behavior of the exact coupled system remains close to the long-run behavior predicted by the invariant-law reduction.

The relevant set distance is one-sided.

Definition 7.4 (One-sided set distance). For nonempty subsets , define

This is the Hausdorff semidistance from to . It measures whether every point of lies near some point of , but it does not require the converse.

That asymmetry is exactly what the reduction theorem can promise. The exact attractor cannot wander far away from the lifted reduced attractor when is small, but the reduced attractor may still contain limiting structures that are not approximated by every nearby exact system without additional hypotheses.

Everything the corollary needs about the exact system is in place from the exact-system theorem of Section 7.1: for every , the exact system generates a continuous semiflow on , the common compact set is forward invariant and absorbing for it with an absorption time independent of , and the omega-limit construction of Chapter 5 therefore supplies the compact global attractor .

Corollary 7.5 (Upper semicontinuity of the exact attractors). Assume the standing finite-state hypotheses of Chapter 2, Assumptions 2.7 and 2.8, Assumption 6.2, and Assumption 7.2, with the invariant-law map supplied by Theorem 6.4. Then for every , the exact semiflow possesses a compact global attractor , and

The mechanism is short and worth seeing directly. We choose a time horizon long enough that the lifted reduced flow sends the absorbing set close to . Then Theorem 7.3 says that on that same horizon, the exact flow is uniformly close to the lifted reduced flow when is small. Invariance of the exact attractor transfers that finite-time estimate to a set-distance estimate.

Proof. Section 7.1 provides, for each , the continuous exact semiflow , the common compact absorbing set , and the compact global attractor .

Fix . Because is the global attractor of the lifted reduced semiflow , there exists such that

Indeed, is the lift of , because depends only on . For lifted sets, the Lipschitz bound of Theorem 6.4 gives

where is the product metric on and is the Lipschitz constant of from the proof of Theorem 7.3. Hence

with the one-sided set distance on , defined as in Definition 7.4. Since attracts the bounded set under , we may choose so that the right-hand side is below .

Now apply Theorem 7.3 with . Since the theorem gives uniform convergence of to over all , there exists such that for ,

Because is invariant and contained in ,

Therefore

Since was arbitrary, the claim follows.

The corollary is the finite-dimensional instance of the standard attractor robustness mechanism; see, for example, Robinson, Infinite-Dimensional Dynamical Systems, Chapter 10, Theorem 10.16 (Cambridge University Press, 2001). We wrote the proof out because in the present setting the argument fits on one page once the tracking theorem is in hand.

Return first to the symmetric Chapter 0 reduction, where the reduced attractor may consist of several equilibria. Corollary 7.5 says that for small , the exact fast-law system cannot acquire late-time states far away from that reduced attractor. The individual equilibria may shift, and their connecting geometry may deform, but the whole asymptotic picture remains close in one-sided set distance.

If one instead uses the asymmetric Chapter 0 variant from Section 0.15 (Breaking the Symmetry), the same theorem has the simpler interpretation that the exact attractor remains close to the lifted reduced equilibrium picture. The point is the same in both cases: the singular perturbation does not create a remote asymptotic branch when the fast law is genuinely fast.

In the routing model, upper semicontinuity says that when browsing dynamics mix quickly, the long-run coupled behavior of policy, critic score, and customer mix stays close to the attractor predicted by the invariant-law reduction. The reduced attractor therefore gives a reliable large-time summary of the exact system in the fast-mixing regime.

This is the right kind of robustness statement for the application. One does not need to classify every equilibrium of the routing dynamics by hand. It is enough to know that the exact asymptotic set sits near the reduced one once the population law equilibrates much faster than the policy moves.

Remark. The corollary uses the one-sided set distance , not the symmetric Hausdorff distance. That is the mathematically correct continuity notion for the theorem we have actually proved: the argument shows that the exact attractor cannot wander far from the reduced attractor, while the converse direction would require additional hypotheses.

The theorem of this section is abstract until we know how to verify Assumption 7.2. That is the job of the next section.

Assumption 7.2 is the right hypothesis for the tracking theorem, but by itself it is still an abstract pathwise stability statement. We now return to the reference-state mechanism from Chapters 2 and 6 and show that it implies the pathwise contraction estimate directly.

The idea is the same as before, but the setting is stronger. In Chapter 6 we froze and used the minorization condition to prove contraction for a single semigroup . In Chapter 7 the actor may move, so we work with a non-autonomous evolution family . Here denotes the solution operator of on all of , so its columns are the solutions started from the basis vectors ; the evolution family of Assumption 7.2 is the restriction of to the zero-mass space , which preserves because it conserves total mass. The same reference-state geometry still works because the lower bound is uniform in : every time block of length sends a definite amount of mass into the same anchor state, no matter how moves during that block.

Proposition 7.6 (Reference-state minorization implies pathwise contraction). Assume there exist a distinguished state and a constant such that

Define

Then, for every measurable path , every , and every , each column of the evolution operator for

dominates . Consequently:

1. if , then Assumption 7.2 holds with

2. if , then , so Assumption 7.2 holds vacuously: every is the zero vector, so the bound reads and any admissible constants serve, for instance and .

In particular, because constant paths are allowed, the same estimate also recovers the frozen contraction mechanism from Chapter 6.

The proof again has three steps. First we prove a one-block lower bound: after time , every basis state sends at least mass into the reference state . Second we rewrite that lower bound as a Dobrushin decomposition of the block propagator. Third we iterate the block contraction on the zero-mass space .

Proof. Fix a measurable path , a scale , and a time . For each initial basis vector , let solve

The simplex-invariance argument from Section 7.1 applies verbatim along the measurable path: total mass is conserved because , and nonnegativity follows from the integrating-factor argument of Section 7.1, whose constant and nonnegative Picard iteration use only that takes values in . Hence remains a probability vector for all , and the columns of are probability vectors. This is the pathwise extension of the Imported fact of Section 6.2, which covers the frozen semigroup .

We first estimate the -component after one block of length .

If , then mass can only leave the reference state at total rate at most , so

Gronwall’s inequality yields

Now suppose . The -component satisfies

so

The reference-state component receives inflow from state at rate at least , while the remaining terms are nonnegative. Hence

Multiplying by the integrating factor gives

integrating from to and using , we obtain

since was chosen exactly so that .

So every column of dominates .

If , then , so the pathwise contraction statement is immediate and there is nothing more to prove. Assume now that . Choose a state . Then

for every , hence

Set

The column lower bound shows that has nonnegative entries. Since both and are column-stochastic, so is .

If , then because the coordinates of sum to zero. Therefore

Every column-stochastic matrix is nonexpansive in . Indeed, if is the positive-negative decomposition of a zero-mass vector, then , and for any column-stochastic matrix ,

Applying this to gives the block contraction

To pass from one block to arbitrary times, note that every propagator is column-stochastic and therefore nonexpansive on . Fix , and write

Iterating the block estimate over the first full blocks and using nonexpansiveness on the remaining interval gives

Finally,

Since , this is exactly

with

Thus Assumption 7.2 holds.

The proposition is the pathwise analogue of Proposition 6.6. The mechanism is unchanged, but the conclusion is stronger: the contraction estimate now holds along every measurable actor path, which is exactly what Theorem 7.3 needs.

Now return to the three-state routing example from Section 2.8. There we had already recorded the crucial structural bounds

for every . In Chapter 2 this was a preview. Here we can cash it out.

Choose the retail hub as the reference state. Then we may take

The total exit rates satisfy

and the second row dominates the other two for every (it is at least , while the first and third rows never exceed and ). The supremum defining is therefore attained in the second row at :

The proof of Proposition 7.6 uses only as an upper bound on the total exit rates, so any upper bound serves in its place, exactly as any lower bound serves for . We may therefore take

Hence

The meaning is the same as in Chapter 6, but now it is a statement about the time-dependent exact law equation. No matter how the actor parameter moves, a uniform fraction of customer mass reaches the retail hub on every block of length . That is enough to force exponential pathwise contraction, and therefore enough to make the tracking theorem applicable.

This is exactly the theorem-level version of the business reading deferred in Chapter 2. “The retail hub remains structurally reachable” is the modeling intuition. Proposition 7.6 is the mathematical statement that turns that intuition into the contraction constants needed by Theorem 7.3.

Remark. The explicit branch in the proposition statement carries real content. When , the zero-mass space is trivial (), and the displayed minorization constants become degenerate. Splitting off the case ensures that the conclusion holds without an implicit appeal to .

At this point the deterministic stationary reduction is complete. Chapter 8 discusses the next two extension directions–non-autonomous forcing and stochastic perturbations–without altering the theorems assembled in this and the preceding chapters.

Exercise 7.1 (Compute: the reduced Chapter 0 system). Start from the asymmetric Chapter 0 model in Section 0.15 (Breaking the Symmetry) and the frozen invariant law . Write the reduced two-dimensional system on explicitly. Verify that the only change is the substitution of by in the actor and critic drifts.

Exercise 7.2 (Verify: the initial-layer integral). In the two-state example, the exact pathwise contraction rate is . Compute

exactly, and use it to derive an explicit version of the law-defect estimate in Theorem 7.3 for the Chapter 0 chain.

Exercise 7.3 (Connect: why the distance is one-sided). In one paragraph, explain why Corollary 7.5 uses the one-sided set distance rather than symmetric Hausdorff distance. Which part of the tracking argument is inherently one-sided?

Exercise 7.4 (Compute: the reduced routing system). Using the explicit formula for from Chapter 6, write the reduced actor and critic equations for the three-state routing model. Identify the exact term in the Section 2.8 formulas where the dynamic customer mix is replaced by the stationary customer mix .

Exercise 7.5 (Break: failure without exponential contraction). Construct a one-dimensional fast variable equation of the form for which the frozen equilibrium exists for each , but the linearization at that equilibrium has zero decay rate. Explain why the proof of Theorem 7.3 breaks down and why an tracking estimate should not be expected.