Efficient Computation of State-Constrained Reachability Problems Using Hopf–Lax Formulae

This article considers two state-constrained reachability problems: computing 1) control-invariant and 2) reach–avoid sets, both under state constraints. Prior research has developed Hamilton–Jacobi (HJ) partial differential equations (PDEs) that characterize the optimal cost functions of these two problems. Unfortunately, solving the HJ PDEs by grid-based methods, such as level-set methods, suffers from exponentially growing computational complexity in the system state dimension. In order to alleviate this computational issue, this article proposes a Hopf–Lax formula for each reachability problem's HJ PDE. The advantage of the Hopf–Lax formulae is that they have more favorable convexity conditions than the corresponding problems. Thus, direct methods may be used to solve Hopf–Lax formulae and thus efficiently compute the optimal solution of the reachability problems under specified conditions. This article provides an example for each reachability problem and demonstrates the performance of the proposed Hopf–Lax method.

kernel [3] is that set of initial states whose SCCIP quantity is less than a certain value.The SCRAP quantifies the distance between the system and a target while the system maintains safety constraints [2].Mathematically, in this problem, we aim to find a control signal to minimize the minimum distance to the goal over a time interval while satisfying state constraints.For example, the reach-avoid set [4] is the set of initial states whose SCRAP quantity is less than a certain value.
Given an HJ PDE, Hopf-Lax theory finds an optimization problem whose optimal cost is the viscosity solution to the HJ PDE [44], [45], [46], [47]; the optimization problem is called a Hopf-Lax formula.Hopf-Lax formulae can significantly reduce computation, as compared to the original HJ PDE, because they do not require gridding the state space [48], [49], [50], [51].Hopf-Lax formulae can be solved by combinations of direct methods (multiple shooting [52] or collocation methods [53]) and numerical optimization methods, such as interior-point methods [54,Ch. 11], sequential quadratic programming [55,Ch. 18], or the split Bregman method [56].As such, for some problems, the computational complexity of solving Hopf-Lax formulae can be polynomial in the state dimension [48].
State-of-the-art Hopf-Lax theory has computed viscosity solutions to HJ PDEs representing state-unconstrained problems [44], [46], [48], [49], [50], [57].On the other hand, there is a small number of papers dealing with HJ PDEs for stateconstrained problems.[22] deals with HJ PDEs whose Hamiltonian has only costate dependency.Lee et al. [51] and Lee and Tomlin [58] present Hopf-Lax formulae for a particular stateconstrained problem for nonlinear systems.Lee and Tomlin [58] present the Hopf-Lax formula for the SCRAP in a time-invariant case: cost, dynamics, and state constraints are time-invariant.In this prior work on Hopf-Lax theory for HJ PDEs relevant to state-constrained problems, there has not been an analysis as to whether the computed results are viscosity solutions.
This article builds on HJ PDEs for the SCCIP and SCRAP problems, which have first been presented in [1] and [2].The current article presents Hopf-Lax theory to compute the optimal cost of these problems whose epigraph is characterized by the viscosity solutions to the HJ PDEs.While the work in [58] first presents the Hopf-Lax formula for the time-invariant SCRAP, it does not present the viscosity solution analysis, and only considers a zero stage cost.
For each of the SCCIP and SCRAP, we consider two cases: 1) a time-varying case where cost, dynamics, and state constraints are time-varying; and 2) a time-invariant case where those are time-invariant [1], [2].These two cases correspond to different Hopf-Lax formulae, resulting in different convexity conditions.Thus, we will choose one of the Hopf-Lax formulae that have more favorable convexity conditions and then apply direct methods to find the optimal cost.On the other hand, the Hopf-Lax formula assumes a convex Hamiltonian in the costate.Since the Hamiltonian in the HJ PDE for the time-invariant SCCIP is nonconvex, there is no Hopf-Lax formula for the time-invariant SCCIP.Thus, this article presents three Hopf-Lax formulae: one for the time-varying SCCIP, one for the time-varying SCRAP, and one for the time-invariant SCRAP.
In summary, our contribution is fourfold as follows.1) We propose three Hopf-Lax formulae for the timevarying SCCIP, for the time-varying SCRAP, and for the time-invariant SCRAP.Previously, Lee and Tomlin [58] first proposed the Hopf-Lax formula for the time-invariant SCRAP, whose stage cost is zero, and this article extends it for nonstage-cost problems.

2) We present the first proof of viscosity solutions from
Hopf-Lax theory relevant to state-constrained optimal control problems.
3) We identify conditions under which direct methods, such as multiple shooting or collocation methods, result in convex problems for the Hopf-Lax formulae but not for the original problems (SCCIP and SCRAP).This article also proves that if the direct methods result in convex problems for the original problems, their Hopf-Lax formulae always result in convex problems.4) We present examples for SCCIP and SCRAP to demonstrate the utility and performance of the proposed Hopf-Lax formulae.The organization of this article is as follows.Section II defines both the SCCIP and SCRAP.Section III presents the HJ PDEs whose solutions' subzero-level sets are the corresponding epigraphs of the SCCIP or SCRAP.Section IV proposes our new theorem in viscosity theory, which is utilized to prove our main results, the Hopf-Lax formulae for the SCCIP, and SCRAP in Section V. Section VI presents convexity analysis for the SCCIP, SCRAP, and their Hopf-Lax formulae.Section VII presents two examples to demonstrate the utility and performance of the proposed Hopf-Lax formulae for each class of problem, and Section VIII concludes the article.

A. Notation
This article uses the subscript * to denote optimality and superscript * to denote the Legendre-Fenchel transformation: α * denotes an optimal control signal, and H * denotes the Legendre-Fenchel transformation of a function H. H * * = (H * ) * denotes the biconjugate of a function H: the Legendre-Fenchel transformation of the Legendre-Fenchel transformation of a function H. "co(B)" denotes a convex-hull operator of a set B.

II. STATE-CONSTRAINED REACHABILITY PROBLEMS AND THE HJ ANALYSIS
This section introduces the two state-constrained reachability problems: 1) the SCCIP and 2) the SCRAP.In these, we are considering the state trajectory (x : [t, T ] → R n ) in a time interval [t, T ] solving the following ordinary differential equation (ODE): where (t, x) are the initial time and state, s is time between t and ( We assume that A is a compact subset in R m .

State-constrained control-invariance problem (SCCIP):
For given initial time and state (t, x), solve where

and
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
τ is the time when the cost is the highest.The scalar function c can handle a number of state constraints.For example, consider state constraints c 1 , . .., c l (s, x(s)) ≤ 0; c = max{c 1 , .., c l } encodes all state constraints into a scalar constraint function.We would like to note that ϑ 1 (t, x) ∈ R ∪ {∞}.This means that, for initial time and state (t, x), if there does not exist a control signal α to satisfy the state constraint (4), the optimal cost ϑ 1 is ∞.Assumption 1. (Lipschitz continuity and compactness): 1) The control set A is compact and convex. 2 Lipschitz continuous in (t, x).7) The stage cost (L) and the terminal cost (g) are bounded ahead.Assumption 1 guarantees the existence of a unique solution to the HJ PDEs, presented in Section III.Notably in which the state constraint c(s, x(s)) ≤ 0 is satisfied in [t, τ ] instead of [t, T ] as in (4) [1].If the optimal value ϑ 1 (t, x) is finite, the state constraint has to be satisfied for all time in [t, T ].
Otherwise, the time maximizer τ will choose the time when the state constraint is not satisfied so that the optimal value becomes infinity.
State-constrained reach-avoid problem (SCRAP): For given initial time and state (t, x), solve where x solves (1), and L, g, f , c are defined in the same way as for the SCCIP.Also, for each (t, x), ϑ 2 (t, x) ∈ R ∪ {∞}, meaning that if there is no feasible control signal to satisfy the state constraint (8), then ϑ 2 (t, x) is infinite.As for SCRAP, we assume Assumption 1 holds.

III. HJ EQUATIONS FOR STATE-CONSTRAINED REACHABILITY PROBLEMS
This section reviews the previously presented HJ PDEs for SCCIP and SCRAP [1], [2], which will be utilized to derive Hopf-Lax formulae in Section V.For each problem, this section presents two HJ PDEs, one for the time-varying case and a second for the time-invariant case.The time-varying case features time-varying cost functions, dynamics, and state constraints.

A. HJ Equation for SCCIP
Lee and Tomlin [1] utilize the epigraphical technique to derive an HJ PDE whose solution characterizes the epigraph of the optimal cost (ϑ 1 ) for the SCCIP.In this formulation, we first encode the cost and constraint of the SCCIP into a state-augmented value function V 1 (10) whose subzero-level set is the epigraph of the optimal cost (ϑ 1 ) for SCCIP epi(ϑ Then, the dynamic programming principle is applied to derive the HJ PDE for V 1 .
Define the augmented value function where x solves (1), (t, x) are initial time and states, and z is a new scalar variable that represents a value axis for the epigraph of ϑ 1 .Now, we will consider (x, z) as an augmented state in V 1 .
V 1 is continuous in (t, x, z)-space, and standard viscosity theory works for V 1 .Theorem 1 presents HJ PDEs for V 1 and finds ϑ 1 from V 1 .
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply. Then

B. HJ Equation for SCRAP
The HJ analysis for SCRAP is similar to that for SCCIP.We first define a state-augmented value function V 2 (17) that combines the cost (7) and the constraint (8) of ϑ 2 so that V 2 's subzero-level set is the epigraph of the optimal cost ϑ 2 for the SCRAP.For (t, x, z) where x solves (1), (t, x) are initial time and states, and z is again the new variable that represents a value axis for the epigraph of ϑ 2 .
Theorem 2 presents HJ PDEs for V 2 and finds ϑ 2 from V 2 .Theorem 2. (HJ PDE for SCRAP [2]): Suppose Assumption 1 holds.V 2 in ( 17) is the unique viscosity solution to the HJ PDE in (0, T ) × R n × R, where H is defined in (12), and For the time-invariant case, V 2 is also the unique viscosity solution to HJ PDE where (12), and Building on the HJ PDEs in this section, Section V presents the main contribution of this article: the proposal of Hopf-Lax formulae for SCCIP and SCRAP.Our Hopf-Lax formulae in Section V will assume convex Hamiltonians in the gradient space.H (12) and HTI 2 (20) are convex in (p, q), but HTI 1 (14) is not.Thus, we do not have the Hopf-Lax formula for the time-invariant SCCIP.Section V presents three Hopf-Lax formulae for time-varying SCCIP, time-varying SCRAP, and time-invariant SCRAP.

IV. VISCOSITY THEORY FOR TWO DIFFERENT PDES' SOLUTION EQUIVALENCE
This section proposes a general theorem in viscosity theory to investigate the equivalence of two first-order PDEs.Building on our new theory in this section, we will present and prove our main result, Hopf-Lax formulae for SCCIP and SCRAP, in Section V. We use a more general notation in this section.
Consider time t ∈ [0, T ], a state that consists of x ∈ R n x , z ∈ R n z , and two value functions and the terminal values for X 1 and X 2 are the same as We say that For SCCIP, F 1 refers to the HJ PDE (11), t to the initial time, x to the initial state, n x to n, n z to 1, and X 1 to V 1 .F 2 refers to the HJ PDE (41) that will be introduced later in Section V-A, X 2 to W 1 (39) also in Section V-A.Also, l(x, z) refers to the terminal condition for V 1 and W 1 : max{c(T, x), g(T, x) − z}.
For SCRAP, we have a similar notation matching rule with SCCIP.
In the notion of viscosity theory, we present conditions under which the two different PDEs F 1 and F 2 have the same solution.Define superdifferentials and subdifferentials of X i (i = 1, 2) with respect to z: for each Theorem 3 states that if two different PDEs (F 1 and F 2 ) are the same in the superdifferential or subdifferential domains in z, the two PDEs' solutions are the same.Theorem 3: Suppose each of the two first-order PDEs in ( 24) with the terminal value (25) Here, r, p, q are the costates with respect to t, x, and z, respectively.Proof: See Appendix A.

V. HOPF-LAX FORMULAE FOR THE STATE-CONSTRAINED REACHABILITY PROBLEMS
For SCCIP and SCRAP, the HJ PDEs in Theorems 1 and 2 can be numerically solved by grid-based methods, such as level-set [23] and fast marching methods [59].These methods require spatial and temporal discretization, which leads to exponential computational complexity in the state's dimension.Thus, it is intractable to utilize these grid-based methods for high-dimensional systems [6].This article provides more discussion about the computational complexity in Section VII-C.This section presents our main results to alleviate this computational complexity, Hopf-Lax formulae for SCCIP and SCRAP: Section V-A for SCCIP, Section V-B for SCRAP, and Section V-C for time-invariant SCRAP.

A. Hopf-Lax Formula for SCCIP
Define, for initial time and state where where p is the costate with respect to x, H is defined in (12), and where b is a new control input (b and p are dual variables).For each s and x, H where ϑ 1 is the optimal cost of SCCIP [(3) subject to (4)], and ϕ 1 is the Hopf-Lax formula for SCCIP [(31) subject to (32)].
We will utilize our proposed theory in Section IV to prove Theorem 4. We first investigate the superdifferentials and subdifferentials of V 1 with respect to z.
Lemma 1. (Convexity of the value function in z): For each and if x, z) is not the empty set, the set of superdifferentials with respect to z is a singleton Note that V 1 is defined in (10).
Proof: See Appendix C. By applying the HJ analysis in Section III-A, we have the following results for the Hopf-Lax formula for SCCIP [ϕ 1 (31) subject to (32)].Combining (31) and (32), define a value function in the augmented state space, where Thus, it is not necessary to add the control constraint in the infimum operation in (39) since H * becomes infinity due to the control constraint (40).
By Theorem 1, W 1 is the unique viscosity solution to Here p and q represents the gradients ∂W 1 ∂x and ∂W 1 ∂z , H * is defined in (34), and Now, it is sufficient to prove We will utilize Theorem 3 to prove V 1 ≡ W 1 .F 1 and F 2 in Section IV refer to (11) and (41), respectively.Also, Lemma 3 analyzes the relationship between the two Hamiltonians: H (12) for V 1 and HW (42) for W 1 .
Lemma 3: where H and HW are defined in ( 12) and ( 42), respectively.Proof: See Appendix D. Now, we are ready to conclude the proof of Theorem 4.

B. Hopf-Lax Formula for SCRAP
Define, for initial time and state where the description for all variables is the same as in Section V-A.Theorem 5. (Hopf-Lax formula for SCRAP): For all where ϑ 2 is the optimal cost of SCRAP [(7) subject to (8)], and ϕ 2 is the Hopf-Lax formula for SCRAP [(46) subject to (47)].
The proof of Theorem 5 similarly follows that of Theorem 4 except for the proof of the convexity of V 2 in z-space, whose counterpart in the previous section is Lemma 1.Although V 2 is convex in z-space, we have to prove it not in a similar way for Lemma 1.Our proof for the convexity of V 2 is added to the proof of Lemma 1 in Appendix B (iii).
C. Hopf-Lax Formula for the Time-Invariant SCRAP Define, for the initial time and state subject to where where p is the costate with respect to x, HTI 2 is defined in (21), H is defined in (33), and where b is a new control input.Theorem 6. (Hopf-Lax formula for the time-invariant SCRAP): Consider the time-invariant SCRAP.For all where ϑ 2 is the optimal cost of SCRAP [(7) subject to (8)], ϕ 2 is the Hopf-Lax formula for SCRAP [(46) subject to (47)], and ϕ TI 2 is the Hopf-Lax formula for the time-invariant SCRAP [(49) subject to (50)].
We will combine Section IV and following formulations to prove Theorem 6.By applying the HJ analysis in Section III-B, we have the following results for the Hopf-Lax formula for the time-invariant SCCIP [ϕ 2 (49) subject to (50)].Combining (49) and (50), define a value function in the augmented state space.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
In (54), we can omit the control constraint in (50): where H TI * 2 is defined in (52).Proof: See Appendix E. By [18,Proposition 3.4], W TI 2 is the unique viscosity solution to where H TI * 2 is defined in (52), and p and q represents the gradients Now, it is sufficient to show V 2 ≡ W TI 2 .To prove this, we will use Theorem 3 where we substitute (20) to F 1 and (56) to F 2 .
Lemma 5 analyzes the relationship between HTI 2 (21) for V 2 and HTI W (57) for W TI 2 .Lemma 5: where HTI 2 and HTI W are defined in (21) and (57), respectively.Proof: See Appendix F. Now, we are ready to conclude the proof of Theorem 6. Proof of Theorem 6: In Theorem 3, we substitute the HJ PDE for V 2 (20) to F 1 and the HJ PDE for W TI 2 (56) to F 2 .Since Lemmas 2 and 5 hold, Theorem 3 implies V 2 ≡ W TI 2 .By Theorem 5, (23), and (58), we conclude ϑ 2 ≡ ϕ 2 ≡ ϕ TI 2 .Although the time-varying case is more general than the time-invariant case, Section VI show that the Hopf-Lax formula for the time-invariant SCRAP has better convexity conditions.Thus, direct methods for the time-invariant SCRAP enable optimality guarantees and efficient computation for more classes of problems than for the time-varying SCRAP.

VI. CONVEXITY ANALYSIS
We can efficiently solve SCCIP, SCRAP, and Hopf-Lax formulae by utilizing direct methods, such as multiple shooting [52] or collocation methods [53].This section investigates convexity conditions under which direct methods guarantee optimality.The convexity analysis in this Section VI is valid regardless of any different numerical integration methods, including the midpoint, trapezoidal, and Simpson's rule, and numerical ODE solving methods, including linear multistep and Runge-Kutta methods [60].For convenience, this section chooses a multiple shooting method that uses the first-order forward Euler method for both numerical integration and ODE solving.
We discretize the time interval to {t 0 = 0, . .., t K = T } where Δ k := t k+1 − t k .In the use of the first-order Euler method, the difference between the exact ODE solution and the Euler approximation becomes smaller for smaller Δ k [61], [62].For notation, the state at t k is x[k], and α[k] and β[k] are control inputs at t k .In this article, we use x to denote both a state trajectory in the continuous-time setting (x(•)) and a state sequence in the discrete-time setting (x[•]).We apply the same notation rule for α, β.
Section VI-A deals with SCCIP and its Hopf-Lax formula, and Section VI-B deals with SCRAP and its Hopf-Lax formulae for time-varying and time-invariant cases.

A. Convexity Analysis for SCCIP and Its Hopf-Lax Formula
This subsection presents a convexity analysis for a temporally discretized SCCIP and Hopf-Lax formula.Consider the temporally discretized SCCIP subject to The temporal discretized SCCIP [(60) subject to (61)] is convex in ) is convex for each k .Thus, the first two conditions in Assumption 2 are sufficient to have convex cost in (60).The other three conditions are for convex constraints (61).

Assumption 2. (Convexity conditions for the temporally discretized SCCIP).
1) We discretize the Hopf-Lax formula for SCCIP [ϕ 1 in (31) subject to (32)] as the following: Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
subject to Based on the above convexity analysis for the temporally discretized SCCIP in b).One of the authors' papers [51] provided sufficient conditions under which a finitedimensional optimization problem (62) subject to ( 63) is convex, as written ahead.
Combining Lemma 6 and the convexity analysis argument for SCCIP, we conclude sufficient convexity conditions for the temporally discretized Hopf-Lax formula for SCCIP in Assumption 3.

Assumption 3. (Convexity condition for the temporally discretized Hopf-Lax formula for SCCIP):
1) L(t, x, a) = L x (t, x) + L a (t, a) for some L x and L a , and x + f a (t, a) for some M and f a for all t ∈ [0, T ].Assumption 3 does not require 1) convex stage cost L in the control input a, 2) affine dynamics f in the control input a, and 3) convex control set A. This is because the Legendre-Fenchel transformation of the Hamiltonian (H * ) is always convex in the control input a, and the convex-hull operator (co) also convexifies the control-input space.We would like to note that the HJ analysis in Section III still needs to assume A is convex, as in Assumption 1.
Remark 1: The Hopf-Lax formula for SCCIP convexifies SCCIP in the control-input space.

B. Convexity Analysis for SCRAP and Its Two Hopf-Lax Formulae
This section presents convexity analysis for SCRAP, the Hopf-Lax formula for the time-varying SCRAP, and the Hopf-Lax formula for the time-invariant SCRAP.The temporally discretized SCRAP is subject to The cost in ( 64) is generally nonconvex in (x[•], α[•])-space since the pointwise minimum operator over k is defined in a nonconvex set: {0, . .., K}.Thus, a sufficient convexity condition is that L and g are 0, as in Assumption 4. In this case, 0 is always a minimizer k .Thus no additional conditions are necessary for convexity of the temporally discretized SCRAP.
The temporally discretized Hopf-Lax formula for SCRAP [(46) subject to (47)] is as follows: subject to By the same argument for SCRAP, H * and g should be zero for the convexity of the discretized Hopf-Lax formula for SCRAP in )-space.[51] proves that L ≡ 0 implies H * ≡ 0, thus, Assumption 4 is also a sufficient condition for convexity of the temporally discretized Hopf-Lax formula for SCRAP.We discretize the Hopf-Lax formula for the time-invariant SCRAP [ϕ TI  2 in (49) subject to (50)] We will prove that Assumption 5 is sufficient for the temporally discretized time-invariant SCRAP to be convex in
Remark 2: SCRAP and its Hopf-Lax formula for the timevarying case are generally nonconvex.However, the Hopf-Lax formula for the time-invariant SCRAP is convex if Assumption 5 holds.Assumption 5 does not require any convexity conditions in the control-input space.

VII. NUMERICAL EXAMPLES
This section provides two numerical examples to demonstrate the Hopf-Lax formulae for SCCIP and SCRAP.For numerical computation, we utilize the interior-point method [54,Ch. 11] in MATLAB, and a computer with an Apple M1 Pro and 16-GB RAM was used.

A. SCCIP: Robust Formation Control
Consider a 16-D system, which consists of four 4-D vehicles: x r (s) = (x r 1 (s), x r 2 (s), x r 3 (s), x r 4 (s)) ∈ R 4 refers to the rth vehicle's 4-D state (r = 1, . .., 4) at time s, and x(s) = (x 1 (s), . .., x 4 (s)) ∈ R 16 refers to the system state at s.Each vehicle has two control inputs: α r (s) = (α r 1 (s), α r 2 (s)) ∈ R 2 .The system dynamics is where x r 1 and x r 2 (x r 3 and x r 4 ) are horizontal (vertical) position and velocity of the rth vehicle, α r 1 is the magnitude of acceleration, and α r 2 is the angle of the acceleration for r = 1, . .., 4. As shown in Fig. 1(a), we would like to control the multivehicle system where the rth vehicle stays close to the point x r g within the interval s ∈ [0, 5] while maneuvering in the rth square-shaped constrained region (S r ).We control the accelerations of the system (70).Due to initial velocities, it is challenging to control the rth vehicle to stay in S r and close to x r g .For this problem, we solve where the target positions are following: The stage cost L is zero, and the terminal cost g is max r=1,...,4 (x r 1 (τ ), x r 3 (τ )) − x r g 2 .For this SCCIP, the Hamiltonian H in (33) becomes where x r i is the name of the variable for which we substitute x r i (s) (r = 1, . .., 4), p r i is the costate with respect to in p, and any supporting hyperplane for H can be written as b • p = 0 for some normal vector b ∈ R 16 .Since the supporting hyperplane b • p = 0 passes through the origin for any b, the Legendre-Fenchel transformation of the Hamiltonian becomes where 16 , and Dom(H * (s, x, •)) is analytically derived by (40).In general, if the stage cost L is zero, H * becomes zero, which has been investigated in [51].
The Hopf-Lax formula for SCCIP in Theorem 1 is subject to where we get the second and third lines in (76) by substituting x(s) into x and β(s) = (β 1 1 (s), . .., β 4 4 (s)) into b in (74).Consider the temporally discretized SCCIP and Hopf-Lax formula for SCCIP on any temporal discretization {t 0 = 0, . . ., t K = 5} as in Section VI-A.The temporally discretized SCCIP is nonconvex, but the temporally discretized Hopf-Lax formula is convex since Assumption 3 is satisfied.Thus, direct methods with numerical optimization methods provide an optimal solution for the proposed Hopf-Lax formula for SCCIP.
For numerical computation of the Hopf-Lax formula, we discretize the temporal space to {t 0 = 0, . .., t 50 = 5} with Δ k = 0.1 (51 time steps).We utilize the interior-point method to solve the temporally discretized Hopf-Lax formula for SCCIP.The computation time is 108.5 s.This system is 16-D with four vehicles, for which it is intractable to utilize grid-based methods (such as the level-set method [23]) to solve the HJ PDEs (11).
As shown in Fig. 1(b)-(h), the four vehicles successfully stay in the constrained regions and stay near their target positions (x r g ).Within the time interval, the maximum distance between x r g and the four vehicles is maximized at 1.1 s as shown in Fig. 1(d).At this time, each distance between x r g and the rth vehicle is 1.18, 0.43 0.42, and 1.16 m, and the first vehicle shows the farthest distance among the four vehicles.Thus, the optimal cost for SCCIP is 1.18.This means that all vehicles stay near their goals while maintaining less than 1.18-m distance within the time horizon while staying in their constraint regions S r for all r = 1, . .., 4. The first vehicle's trajectory shown in Fig. 1(b)-(h) initially moves downward, with a velocity in the -y-direction.Thus, in the solution found by SCCIP, the optimal control first decreases its vertical speed to ensure that it remains in S 1 .Then, it heads toward x 1 g , in order to stay close to x 1 g , as incentivized by the cost function.
This example shows an optimal-control analysis for SCCIP, and this can be additionally utilized in decision-making for hardware specification.For example, consider designing multiple mobile manipulators for which we attach a manipulator to each mobile robot (vehicle).Each manipulator aims to perform some tasks at its target position x r g for all time in [0,5].Our SCCIP analysis provides a guideline for choosing the manipulator's workspace that has to cover more than 1.18 m since all vehicles can be controlled to stay within 1.18-m-radius regions from x r g for all time in [0,5].
As shown in Fig. 2(a), we would like to control the multivehicle system so that the distance between the 2-D center position of the six vehicles and the goal point is minimized while maintaining the hexagonal formation with 0.1-error bound where x g = (1, 0), and d r = 0.5(cos((r − 1)π/3), sin((r − 1)π/3)) is a vector for the rth vehicle that indicates the formation direction from the 2-D center position of the six vehicles (r = 1, . .., 6).
For this problem, we will derive the Hopf-Lax formula for the time-invariant SCRAP since the temporally discretized Hopf-Lax formula for the time-varying SCRAP as in Section VI-B is generally nonconvex.
The Hamiltonian H TI 2 (x, p) (51) is where x = (x The Hopf-Lax formula for the time-invariant SCRAP in Theorem 6 is Consider the temporally discretized SCRAP and Hopf-Lax formula on any temporal discretization {t 0 = 0, . .., t K = 5} as in Section VI-B.The temporally discretized SCRAP is nonconvex, but the temporally discretized Hopf-Lax formula for the time-invariant case is convex since Assumption 5 is satisfied, which enables direct methods to compute an optimal solution. We utilized a numerical algorithm presented in [58] to find optimal control signals and state trajectories for the SCRAP problem.The computation time is 158.5 s.Fig. 2(b)-(h) shows the optimal state (position) trajectories of the multivehicle system.The distance between the center of the six vehicles and the goal position x g is minimized at 2.3 s, as shown in Fig. 2(e), and then, the center of the six vehicles moves away from the goal position.
This example demonstrates our method's potential usefulness in multiple vehicle operations, for example, the control a spacecraft system to get close to an object of interest, such as an asteroid, and then the release of smaller exploration robots from the spacecraft to do more detailed sensing of the asteroid.The SCRAP analysis provides the optimal control and the time at which the exploration robots have to be released from the spacecraft.

C. Computation Time
In this section, we compare the computation time of the Hopf-Lax formula for the time-invariant SCRAP and the level-set method.In the example in Section VII-B, we can easily change the number of vehicles.Fig. 3 shows the computation time where the number of vehicles is 1-10.We used the interior-point method [54,Ch. 11] to solve the proposed Hopf-Lax formula, which has polynomial-computational complexity in the state dimension, which is also demonstrated in Fig. 3. Since each vehicle is 2-D, the state dimension varies from 2 to 20.
The level-set method requires spatial and temporal discretization, which leads to exponential computational complexity in the state's dimension.The level-set method handles this problem if the number of vehicles is smaller than three.For numerical computation, we discretize the temporal space with 51 points and each axis of the augmented state (x, z) space with 61 points.The computation time is 26.9 s for the one-vehicle and 27.4 h (9.86 × 10 4 s) for the two-vehicle.
The level-set method provides a closed-loop control.Thus, offline computation is allowed because a closed-loop control is robust to disturbances, measurement noise, and system modeling errors in practice.On the other hand, our Hopf-Lax methods provide an open-loop control, requiring real-time computation to be robust to the above factors.
Offline computation of the level-set method is intractable for high-dimensional systems due to the exponential growth of computational complexity in the state dimension and computing machines' memory limits.However, our Hopf-Lax method computes a solution even if the computation time might not meet real-time computation.For the three-vehicle setting (6-D), the level set method requires 641.1 TB of memories for a numerical solution to the HJ PDE, where 51 temporal discretization points and 61 spatial discretization points on each axis of (x, z) are used.It still requires 367.4 gigabytes if the number of spatial discretization points on each axis is reduced by 21.On the other hand, our Hopf-Lax method computes a solution even for a 10vehicle setting (20-D).In order to reduce the computation time, the Hopf-Lax formula can be incorporated with approximation methods, such as the receding-horizon technique [63], [64].

VIII. CONCLUSION
This article considers two state-constrained reachability problems: 1) the control invariance problem (SCCIP) and 2) reachavoid problem (SCRAP).We propose three Hopf-Lax formulae for 1) SCCIP, 2) SCRAP, and 3) the time-invariant SCRAP and provide proofs.In the proof of each Hopf-Lax formula, this article considers two HJ PDEs: One for SCCIP (or SCRAP) and the other for the corresponding Hopf-Lax formula and then proves these two different HJ PDEs have the same solution.This proof technique can be generally applied to other state-constrained optimal control problems' Hopf-Lax theory.This article also shows that Hopf-Lax formulae always have more favorable convexity conditions than SCCIP and SCRAP.Thus, direct methods efficiently compute solutions without losing optimality guarantees for more classes of problems.This benefit has been demonstrated via simulations compared with HJ analysis.The level-set method to solve HJ PDEs requires exponentially increasing memory space and computation time in the state dimension, so it is intractable to deal with more than 5-D systems.On the other hand, our method deals with 20-D and potentially much higher-dimensional systems without losing the optimality guarantee under the specified conditions.

A. Proof of Theorem 3
i) The terminal values of X 1 and X 2 are the same.ii) Consider a smooth function U : Since X 1 − U has a local maximum at (t 0 , x 0 , z 0 ), ∂U ∂z (t 0 , x 0 , z 0 ) is in ∂ + z X 1 (t 0 , x 0 , z 0 ).Then, by ( 30) iii) Consider a smooth function U : . By ( 30)

B. Proof of Lemma 1
i) For all y 1 , y 2 , y 3 , y 4 ∈ R, The second inequality is by (89).For α ∈ A, we use τ * (α) to denote a maximizer of the last term in (92).By the triangular inequality, we simplify (92) to Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
The last inequality holds by the definition of V 1 in (10).iii) We additionally prove that V 2 (17) is convex in z-space: Since the last term in (94) is greater than or equal to θ 1 V 2 (t, x, z 1 ) + θ 2 V 2 (t, x, z 2 ) by the triangular inequality, we conclude the proof.

C. Proof of Lemma 2
i) The proof of (37).
ii) The proof of (38).The convexity of V 1 (t, x, z) stated in Lemma 1 implies that ∂ + z V 1 (t, x, z) contains a single superdifferential q if V 1 (t, x, z) is locally affine in z, otherwise, ∂ + z V 1 (t, x, z) is the empty set.If V 1 (t, x, z) is locally affine in z, it is also differentiable in z.As z converges to 0 in (95), we have iii) We would like to note that the analogous derivation concludes that this lemma also holds for V 2 : ∂ − z V 2 (t, x, z) ⊂ [−1, 0] for all (t, x, z) ∈ [0, T ] × R n × R, and if ∂ + z V 2 (t, x, z) is not the empty set, then ∂ (103 Since H is convex in p for each (t, x) and lower semi-continuous in p, H * * ≡ H. Thus, we have HW (t, x, z, p, q) = −q max = −qH * * t, x, − p q = −qH t, x, − p q .(104) By ( 103) and (104), we conclude H(t, x, z, p, q) = HW (t, x, z, p, q) (105) for all q < 0.

E. Proof of Lemma 4
This proof generalizes the proof of Lemma 1 [58], which is for the zero stage cost problem.

Fig. 3 .
Fig. 3.This figure shows the computation time of the level-set method and the time-invariant Hopf-Lax formula with different numbers of vehicles.Each vehicle is 2-D.
since this control constraint is the domain of the control input β(s) for finite H TI * 2 (x(s), •).Lemma 4. (Domain of H TI * 2 Assumption 5 holds.According to the definition of H TI 2 (51), H TI 2 does not have state dependency since f and L do not have it.Thus, H TI * 2 also does not have state dependency and is always convex in Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
∈ R 2 , and p = (p 1 , . .., p 6 ) ∈ R 12 .Since H TI 2 is a pointwise maximum of two convex functions in p, H TI 2 is convex in p.Also, for all b ∈ R 12 , the supporting hyperplane of H TI 2 in p-space with respect to the normal vector b crosses the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.