Chapter 3: Local Lipschitz Regularity and Well-Posedness

Chapter 2 ended with a precisely stated system: the prescribed-closure ODE (L1) on the enlarged phase space , with a softmax policy, a uniformly coercive critic, a damped actor, and a prescribed Lipschitz closure map. The Chapter 0 example showed what the formulas look like when every variable is a scalar, and Chapter 2 confirmed that the general definitions recover that example exactly.

A formula for the vector field is not yet a well-defined evolution. We first need a local ODE statement: for initial data in , the ambient ODE determined by the same formulas should have a unique solution for some positive time, and that solution should depend continuously on the initial data. These conclusions follow from the Picard-Lindelöf theorem once the vector field is locally Lipschitz on an open neighborhood.

This chapter proves that condition. We check, component by component, that the softmax policy, the actor drift, the critic drift, and the law field are all locally Lipschitz on the phase space and on the ambient Euclidean space. For the softmax, actor, and critic components, the strategy is the same: the defining formulas are built from smooth operations–exponentials, finite sums, products, polynomials–so these pieces are , hence locally Lipschitz. The law field is the one exception, because it contains the closure map , which is assumed Lipschitz rather than . Once all four components are verified, the full vector field inherits the same regularity, and the Picard-Lindelöf theorem gives local existence and uniqueness.

What this chapter does not prove is that solutions exist for all positive time. Local existence guarantees a solution on some interval , but the maximal existence time could be finite if the solution escapes every compact set. Proving that requires the a priori estimates of Chapter 4: the actor stays in its box, the distribution stays on the simplex, and the critic remains bounded.

Every Lipschitz argument in this chapter rests on a single general fact, which we record here for reference.

Recalled fact (Differentiability implies local Lipschitz continuity). Let be a continuously differentiable map on an open set . Then is locally Lipschitz on : for every compact convex set , one has

where and is the operator norm of the Jacobian matrix. This is just the mean value inequality: if , then and

where denotes the line segment from to . For a general compact set , enclose it in a closed ball : when is on all of , as it is in every application below, the principle applied to the compact convex set gives the same bound for , with the supremum now taken over .

The consequence for us is that to show a component of the vector field is locally Lipschitz, it suffices to show that it is . Since all our formulas are built from exponentials, logarithms of strictly positive quantities, polynomial functions, and finite sums, checking the property will be straightforward. The one exception is the law field, which involves the prescribed closure map . Because is only assumed Lipschitz (not ), the law field requires a direct Lipschitz estimate rather than a differentiability argument. We will handle that case separately in Section 3.4.

The vector field of system (L1) involves the softmax policy in three ways: through the occupancy measure (which multiplies by the distribution), through the logarithm in the entropy term of the actor drift, and through the score function that sets the direction of the actor update. The question is whether these three maps are regular enough for the Lipschitz theory.

All three are smooth: they are functions of on all of . The reason is that the softmax denominator is always strictly positive, so no division by zero ever occurs.

Lemma 3.1 (Softmax regularity). For every and , the maps

are on . In particular, they are locally Lipschitz on , and on any compact set , each map is Lipschitz.

Proof. The softmax policy is defined by

The numerator is an exponential of a linear function of , hence smooth. The denominator is a finite sum of such exponentials, hence also smooth. Because each exponential is strictly positive, the denominator is bounded below by the positive quantity for any single term, and in particular the denominator is strictly positive for every . A ratio of smooth functions with a strictly positive denominator is smooth. Therefore is on .

Since everywhere, the logarithm is the composition of the smooth function with the smooth function on . Therefore is .

The score function is the gradient of a smooth function, hence smooth: is .

By the regularity principle stated above, each of these smooth maps is locally Lipschitz on and Lipschitz on any compact subset.

Quantitative bounds on the actor box. Although smoothness alone suffices for the Lipschitz property, the explicit bounds on will be useful later. Write

Since on , every logit satisfies . Therefore for every pair . The softmax numerator is at least , the denominator is at most , and so on one has:

The first bound is the lower bound just obtained. The logarithm bound follows from it, and the score bound is the Chapter 2 formula whose right-hand side has norm at most . These bounds are finite and depend only on the data. The gradient bound shows that is Lipschitz on with constant at most , and hence so is , since . The Lipschitz constant of the score map on is finite by Lemma 3.1, because the second derivatives of are continuous on the compact set .

Verification in the Chapter 0 example. In Chapter 0, the softmax at state for action reduces to . Its derivative is

where . The product is maximized at where , giving . On the actor box , one therefore has , which means the Lipschitz constant of on is .

The score function at is . Its derivative is , which satisfies on . The Lipschitz constant of the score on is therefore at most .

The general bound gives , consistent with the fact that the score lies in .

Takeaway. The softmax, its logarithm, and its score function are smooth in because the softmax denominator is always positive. On any compact set, each of these maps is Lipschitz with a finite, data-determined constant. In the Chapter 0 example, the softmax reduces to the logistic sigmoid with Lipschitz constant on .

The actor drift combines several ingredients: the occupancy , the advantage-like signal (reward, critic contribution, and entropy), the score function, and the boundary damping . The question is whether the resulting map is locally Lipschitz.

For the dynamical-systems reader. The particular structure of the actor drift comes from the policy-gradient update rule in reinforcement learning. The occupancy weights each state-action pair by how often the agent visits it. The score function points in the direction that increases the probability of action at state . The advantage-like signal combines three forces: the immediate reward, the critic’s estimate, and an entropy-regularization penalty. The product of these three ingredients is summed over all state-action pairs, and the result is multiplied by the damping matrix that confines the actor to its box. The key structural point is that every factor in this sum is a smooth function of , which is why the actor drift is smooth.

Lemma 3.2 (Actor drift regularity). The maps

are on . In particular, they are locally Lipschitz, and on any compact subset they are Lipschitz.

Proof. Recall the definition of the raw actor drift:

We check the regularity of each factor in this sum.

The proof has three steps: identify each building block, verify its regularity, and conclude by the closure of under algebraic operations.

Step 1 (Building blocks). The sum involves five types of building blocks:

1. : a product of the coordinate function (linear in , hence smooth) and (smooth in by Lemma 3.1). The product is smooth in and does not depend on .

2. : a constant.

3. : linear in , hence smooth.

4. : smooth in by Lemma 3.1.

5. : smooth in by Lemma 3.1.

Step 2 (Closure under algebraic operations). The product of two maps is (by the product rule). The sum of finitely many maps is . Each term in the double sum over is a product of the five building blocks listed above, all of which are in . Therefore each term is , and the finite sum is on .

Step 3 (Damping). The damping matrix is a polynomial in , hence smooth. The product is therefore on .

By the regularity principle, is locally Lipschitz and Lipschitz on any compact subset.

An important observation about the ambient domain: in Lemma 3.2, the maps are on all of , which is wider than the phase space . The formulas involve , which makes sense for any ; no positivity or normalization of is needed. This algebraic extension to the full Euclidean space is essential in Section 3.5 when we apply the Picard-Lindelöf theorem.

Verification in the Chapter 0 example. In the symmetric Chapter 0 model, the actor drift is

As in Chapter 2 (Section 2.7), the absence of from this formula is the antisymmetric-feature accident of the symmetric example, not a general property, and the cancellation uses the simplex identity . Under the full two-coordinate ambient extension (Definition 3.5 below), the raw drift carries the prefactor , which equals only on the simplex; in the reduced coordinates used throughout Chapter 0 and in Exercises 3.2-3.3, the formula above is its own ambient extension, unchanged for arbitrary .

Each factor is smooth: is a polynomial, is the logistic sigmoid (smooth), is smooth, and is affine. The product of smooth functions is smooth, so is on , and therefore locally Lipschitz.

To get an explicit bound on the Chapter 0 absorbing set (the set , not the action count), we can compute the partial derivative with respect to :

On , one has and , so . We defer the full Jacobian computation, including the -derivative and the -derivative (which vanishes in the reduced symmetric model), to Exercise 3.3. The Lipschitz constant of on is the supremum of the Jacobian norm over , which is finite because is and is compact.

Takeaway. The raw actor drift and the damped drift are on the full ambient space because they are built from smooth building blocks (softmax, score, polynomial damping, linear terms) combined by products and sums. The formulas make sense for any , including vectors that are not probability distributions.

The critic equation is the simplest component to analyze, because it is linear in the critic variable and smooth in the remaining variables.

Lemma 3.3 (Critic drift regularity). The maps

are on , and the full critic drift

is on .

Proof. Recall the definitions from Chapter 2:

Each occupancy term is in (smooth in by Lemma 3.1, linear in ). The constants , , and are data. Finite sums and products of maps with constants are . Therefore and are on .

The map is the product of a matrix-valued function and the vector , hence . The difference is therefore on , and by the regularity principle it is locally Lipschitz and Lipschitz on any compact subset.

Verification in the Chapter 0 example. In Chapter 0, the critic drift is , a scalar function of . Its partial derivatives are

On the absorbing set , the first partial is bounded by , and the second is bounded by . Both are finite, confirming that the critic drift is Lipschitz on .

Takeaway. The critic drift is in because it is linear in with coefficients. This is the simplest of the four components: the linearity in means no exponentials or logarithms appear in the critic direction.

The law field involves the prescribed closure map , which is only assumed to be Lipschitz, not necessarily differentiable. The regularity principle ( implies locally Lipschitz) therefore does not apply directly. Instead, we estimate the Lipschitz constant by hand.

Recall from Chapter 2 (Definition 2.9) that the closure map is defined first on the actor box, and that is its Lipschitz constant there, measured in the -norm. The Picard-Lindelöf argument in this chapter compares points in the ambient Euclidean space, so we use the clipped extension where is the coordinatewise clamp. The clamp is -Lipschitz, hence is Lipschitz on with the same constant , it takes values in , and it agrees with on the phase space. This also absorbs the running affine example: is simplex-valued on the actor box used in the model; the ambient field uses its clipped extension. To keep notation light, the estimates below write for this ambient extension.

Lemma 3.4 (Law field regularity). The map

is Lipschitz on (globally, not just locally), with Lipschitz constant at most .

Proof. The map does not depend on , so we only need to estimate the variation in . For any two points and in ,

where the last inequality uses and the fact that the law field does not depend on .

The constant is attained under the sum norm , the same product norm fixed in Section 3.5 for the ambient space, provided is the optimal Lipschitz constant of : perturbing alone realizes the factor , and perturbing alone realizes the factor . Up to a uniform factor (norm equivalence in finite dimensions), the bound holds for any choice of norm.

The law field is the only component that is globally Lipschitz (with a uniform constant on all of ), because it is affine in and the closure map is globally Lipschitz. By contrast, the actor and critic components are only locally Lipschitz: their Lipschitz constants grow as one moves to larger compact sets.

Verification in the Chapter 0 example. The Chapter 0 closure map is , which is Lipschitz with constant in the -norm on (as computed in Exercise 2.4: each component moves by , and the -norm adds the two contributions). The relaxation rate is , so Lemma 3.4 gives the global Lipschitz constant . In the reduced coordinate used throughout Chapter 0, the law field is

and its Lipschitz constant on (in the variables ) is at most , where is the slope of the single retained component of ; the bound matches the ambient constant, since

Takeaway. The law field is globally Lipschitz because the closure map is Lipschitz and the relaxation is affine in . This is the only component of the vector field with a uniform Lipschitz constant on the entire ambient space.

The four lemmas above establish that each component of the vector field is well-behaved. We now assemble them into the full vector field and apply the classical existence and uniqueness theorem.

For the RL reader. The result of this section gives the “mean dynamics” the status of a genuine ODE flow: once the ambient vector field is locally Lipschitz, every starting configuration of actor, critic, and distribution generates a unique local trajectory, and nearby initial data generate nearby trajectories for as long as those solutions coexist. Later questions about invariance, absorbing sets, and attractors rely on this local well-posedness.

Why the ambient space matters. The phase space is the natural domain for the dynamical system: the actor lives in the compact box , and the distribution lives on the simplex . But is not an open subset of Euclidean space, since both and have boundaries. The Picard-Lindelöf theorem, in its standard form, requires the vector field to be defined on an open set. We therefore need to extend the vector field from to an open neighborhood.

A common textbook approach for extending a vector field from a closed domain to an open set is to multiply by a smooth cutoff function that equals on the domain and has compact support in an ambient neighborhood. That approach works in general, but it is unnecessary here: the defining formulas for the vector field already make sense on all of . The softmax is defined for every , because the softmax denominator is strictly positive there. The occupancy formula makes algebraic sense for any , even when is not a probability vector. The damping matrix is a polynomial and extends to all of . The closure map is defined on all of . We therefore choose the algebraic ambient extension over a smooth cutoff: no restriction to is needed, the formulas are their own extension, and the exact structure of the vector field is preserved outside .

Definition 3.5 (Ambient vector field). Define by

where for arbitrary , and all other formulas (Definitions 2.1 and 2.3—2.6, together with 2.9—2.10) are applied unchanged.

On , we use the product norm where the -component carries the -norm, as in the Chapter 2 metric , which this norm extends. Since all norms on a finite-dimensional space are equivalent, the local Lipschitz property and the form of the Lipschitz constant up to a uniform factor are independent of this choice; we fix the sum norm so that the global constant in Lemma 3.4 takes the clean form .

On the phase space , the ambient extension agrees with system (L1) of Definition 2.10; the extension merely removes the restriction that , , as discussed in the algebraic-extension paragraph above.

Proposition 3.6 (Local well-posedness of the ambient ODE). The ambient vector field is locally Lipschitz on . For every compact set , there exists such that

Consequently, for every initial condition , the Cauchy problem

has a unique maximal solution , with . If , then the solution leaves every compact subset of as .

Proof. The proof assembles the four component lemmas and applies the Picard-Lindelöf theorem.

Step 1 (Local Lipschitz of the full field). The ambient vector field has three components:

• , which is by Lemma 3.2;
• , which is by Lemma 3.3;
• , which is (globally) Lipschitz by Lemma 3.4.

A map with components is , hence locally Lipschitz. A map with one Lipschitz component and two components is locally Lipschitz: on any compact set , the first two components are Lipschitz (by the regularity principle applied to ), and the third is Lipschitz (globally). Therefore

for all , where , , are the Lipschitz constants of the three components on , and the sum norm fixed above is in force.

Step 2 (Picard-Lindelöf). We now import the following classical statement, which packages two standard results of local ODE theory: local existence and uniqueness under a Lipschitz condition (Picard-Lindelöf), and the continuation of solutions to a maximal interval with the compact-escape alternative. Both are proved in [Coddington and Levinson, Chapter 1].

Theorem (Picard-Lindelöf / Cauchy-Lipschitz, with continuation). Let be locally Lipschitz on an open set . For every , the initial-value problem , , has a unique maximal solution . If , then leaves every compact subset of as .

Take (which is open) and (which is locally Lipschitz by Step 1). The theorem gives a unique maximal solution for every initial condition in , in particular for every initial condition in .

The blow-up alternative states: if , the solution must leave every compact set. Equivalently, if the solution remains in a compact subset of for all , then . This is the gateway to global existence, which Chapter 4 will open by proving that the solution stays in a compact set.

The same Picard-Lindelöf circle of ideas, via a Gronwall comparison of two solutions, also gives continuous dependence on initial data in the usual local sense (this too is proved in [Coddington and Levinson, Chapter 1]): if is an initial point and , then there is a neighborhood of such that the solutions starting from all exist on , and the map is continuous on . Once Chapter 4 proves that solutions starting in stay in and exist for all , these local solution operators will assemble into the continuous semiflow of Definition 1.1.

Takeaway. The ambient vector field is locally Lipschitz on the full Euclidean space because the softmax, actor, and critic components are and the law field is Lipschitz. The Picard-Lindelöf theorem then gives a unique maximal ambient solution for every initial condition in . Chapter 4 will prove that these trajectories remain in the phase space and extend for all positive time.

Proposition 3.6 guarantees that the ambient ODE has a unique maximal solution from every initial condition in , but only on a maximal interval that might be finite. A finite would mean that the ambient trajectory leaves every compact subset of ; because the ambient space is all of Euclidean space, this is equivalent to norm blow-up. For the prescribed-closure system to define a semiflow on , we need two further facts: solutions starting in must remain in , and their maximal existence time must be infinite.

The blow-up alternative from Picard-Lindelöf says: if the solution stays in a compact set for all , then . The strategy for proving global existence is therefore to find a compact set that the solution cannot escape.

In system (L1), three mechanisms work together to prevent blow-up:

1. Actor confinement. The damping matrix vanishes on the boundary of , so the actor loses its outward motion at each face of the box . Chapter 4 turns this mechanism into a coordinatewise barrier argument: on each face , the -th coordinate of the actor drift vanishes, so if then for all .

2. Simplex invariance. The distribution equation has an explicit variation-of-constants formula showing that if and for all , then for all . This keeps the distribution in the compact simplex .

3. Critic coercivity. The matrix satisfies for every and every (Chapter 4, Proposition 4.3); coercivity can fail for outside the simplex, which is why actor confinement and simplex invariance must be established before the critic estimate is applied. On , this uniform coercivity gives an exponential decay estimate for , which prevents the critic from diverging in finite time.

Once these three facts are established, the solution stays in the compact set

for a suitable depending on the initial critic norm, and the blow-up alternative forces .

Preview from the Chapter 0 example. In the worked example, the absorbing set was . The actor stays in because of the damping . The distribution stays in because the relaxation equation preserves the simplex. The critic satisfies where , because of the coercivity bound. Together, these confine every trajectory to a compact set, and Picard-Lindelöf extends local solutions to all positive time.

The details of these three confinement arguments, and the explicit absorbing set construction that follows, are the content of Chapter 4.

Takeaway. Local existence (this chapter) and a priori confinement plus invariance (Chapter 4) together yield global trajectories that stay in . At that point the time- solution operators form the continuous semiflow of Chapter 1.

This chapter has established the regularity of the vector field in system (L1) and the local well-posedness of its ambient ODE extension. The main results are:

• The softmax policy, its logarithm, and its score function are smooth in the actor parameter , because the softmax denominator is always strictly positive (Lemma 3.1).

• The actor drift is on the full ambient space, because it is built from smooth building blocks combined by products, sums, and polynomial damping (Lemma 3.2).

• The critic drift is , because it is linear in with coefficients (Lemma 3.3).

• The law field is globally Lipschitz, because the prescribed closure map is Lipschitz and the relaxation is affine in (Lemma 3.4).

• The full ambient vector field is locally Lipschitz on all of , and the Picard-Lindelöf theorem gives a unique maximal ambient solution for every initial condition in (Proposition 3.6).

The reader can now verify, for the Chapter 0 system, that each component of the vector field is smooth and compute the Lipschitz constant on the absorbing set (Exercise 3.3).

What remains is to show that solutions starting in stay in the phase space and extend to all positive time. That step requires three a priori estimates–actor confinement, simplex invariance, and critic coercivity–which together trap trajectories in a compact set and force the maximal existence time to be infinite. These estimates are the subject of Chapter 4, which will also construct the compact absorbing set and complete the semiflow.

Exercise 3.1 (Lipschitz constant of the softmax). Compute the derivative of and show that . Using the fact that , determine the Lipschitz constant of on . Compute the Lipschitz constant of the score function on the same interval.

Exercise 3.2 (Ambient extension in the Chapter 0 model). Write the Chapter 0 vector field as a map from to , without assuming or . Verify that for , the ambient extension agrees with the prescribed-closure system of Chapter 0. Evaluate the ambient vector field at the point –a point outside –and confirm that the formulas are well-defined.

Exercise 3.3 (Lipschitz constant of the full vector field). Compute the Jacobian matrix of the Chapter 0 vector field at a general interior point of the absorbing set . Find the maximum of the operator norm over (a numerical estimate is sufficient). This gives the Lipschitz constant of on .

Exercise 3.4 (Effect of removing the damping). Set for all (removing the boundary damping) and consider the resulting undamped actor drift .

(a) Is the undamped vector field still locally Lipschitz on ?

(b) If yes, does the Picard-Lindelöf theorem still apply to give local existence?

(c) What property that the damped system has does the undamped system lose? Explain why this matters for the global existence argument.

Exercise 3.5 (Why local Lipschitz?). The Picard-Lindelöf theorem requires local Lipschitz continuity of the vector field to guarantee uniqueness of solutions. Construct an example of a scalar ODE where is continuous but not locally Lipschitz, and the ODE has more than one solution from a single initial condition. (Hint: look for a power-law nonlinearity with .)

Exercise 3.6 (Infinite-dimensional extension). System (L1) lives on a finite-dimensional phase space because the state space is finite. If were a compact metric space (infinite state space), the distribution would be a probability measure on , and the phase space would be , where is the space of Borel probability measures with the weak topology. State what additional property of the vector field (beyond local Lipschitz continuity) would be needed to apply a well-posedness theorem in this infinite-dimensional setting. (Hint: the Picard-Lindelöf theorem extends to Banach spaces, but the compactness arguments that give the absorbing set and the attractor require additional structure.)