Gain-Scheduled Mean-Field-Type Control for a Non-Linear Continuous Stirred Tank Reactor

We present the design of a Gain-Scheduled Mean-Field-Type Control (GS-MFTC) for a non-linear Continuous Stirred Tank Reactor (CSTR). The MFTC controllers belong to a class of stochastic control techniques considering the distribution of the system states, and control inputs, while minimizing the variance of both variables of interest. The underlying risk-aware control problem is solved explicitly for the infinite horizon in order to determine the required gains composing the feedback control input law. We present both the risk-neutral approach involving the first and second moment (mean and variance terms), and the risk-sensitive approach involving higher-order terms.


I. INTRODUCTION
Mean-Field-Type Control (MFTC) strategy is a stochastic technique that considers the distribution of the system states and control inputs in both the cost functional and the system dynamics [1]. In this regard, the MFTC controllers incorporate mean and variance terms, becoming risk-aware approaches as in the portfolio paradigm introduced in [2], i.e., the MFTC controllers minimize risk terms in either the context of continuous time as studied in [3]- [5], or in discrete time as studied in [6]- [8]. The mean-field-type approach has been recently addressed from a game-theoretical perspective. Mean-Field-Type Games (MFTG) have been mainly studied in continuous time by using different methods, e.g., Stochastic Maximum Principle [4], [9], [10], Dynamic Programming Principle [11], Wiener Chaos Expansion [12], and the Direct Method [13]- [20]. Besides, the study of non-linear scenarios is quite appealing in the engineering field in order to enlarge the spectrum of problems that can be addressed by means of these techniques. For instance, in [21], it has been shown that the direct method is also suitable for solving non-quadratic non-linear MFTG.
The aforementioned works study the theoretical solutions for the mean-field-type game and control problems. However, The associate editor coordinating the review of this manuscript and approving it for publication was Roberto Sacile.
there is an increasing need to implement these results in engineering applications. For instance, in [22], price dynamics are analyzed in the context of mean-field-type games. In [23] and [24], a risk-aware MPC controller is designed for drinking water networks, and in [25], blockchains are analyzed from a mean-field-type perspective. In this work, we present another implementation of the mean-field-type theory into a non-linear engineering system for the design of a riskaware controller.
On the other hand, an alternative to control a non-linear system is the gain scheduling technique [26], which consists of decomposing the design of a non-linear controller into a set of different linear controllers. Thus, the design of multiple linear controllers can be used to stabilize a non-linear system. In order to motivate the need of strategies and methodologies to control non-linear systems, we present an industrial system that has been used to illustrate and evaluate the performance of several non-linear control techniques, i.e., the Continuous Stirred Tank Reactor (CSTR), whose deterministic version has been presented in [27]. Here, we use the non-linear stochastic model. For instance, in [28], a distributed model predictive is designed for a linear CSTR. In [29], a nonlinear adaptive control is designed by using Fourier integral for a CSTR. In [30], a robust gain-scheduled PI control is designed for a CSTR inspired from the model and robust control presented in [31]. VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ As a contribution, and different from other works that have addressed the control design for the non-linear CSTR, the aim of this work consists in presenting the design of risk-aware controllers, which are different from either the classical Linear-Quadratic-Regulator (LQR) or Linear-Quadratic-Gaussian (LQG) controllers (see Table 1). The contributions of this paper are summarized as follows: • We introduce the analysis of both quantification and minimization of risk in the design of optimal stochastic controllers.
• We study the risk-neutral case involving the first and second moments, and the risk-sensitive approach including higher order terms [32, Remark 2].
• Different from the existing literature where linear and risk-free problems are considered, we design a nonlinear risk-aware controller. Besides, to the best of our knowledge, classical linearization-based gain scheduling technique has not been implemented yet to design risk-aware controllers.
• Different from the work presented in [13], where game problems are discussed for the finite horizon, we design a risk-aware controller for the infinite time horizon. Thus, the appropriate gains that determine the feedback optimal control law in terms of the mean and the variance of the system states are computed, obtaining a MFTC for a stochastic and non-linear system. This work also serves as an applicable illustration of recent theoretical results.
• We discuss the differences between the well-known LQG and the proposed risk-aware controller, showing that the latter controller uses a more sophisticated feedback structure in order to minimize the variance for both the system states and the control inputs (see Table 1).
• We study the local stability of the different operational points for the closed-loop system involving the proposed risk-aware MFTC controller.
• The CSTR system has been extensively studied in the mean-field-free case to illustrate the performance of optimal control strategies. We present the design of a risk-aware controller involving mean-field-type terms for a CSTR, i.e., we present the GS-MFTC in a stochastic CSTR where it is required to control both the reactant concentration and the reactor temperature by minimizing the risk. This paper is organized as follows. Section II presents a stochastic version of the CSTR. Section III presents the linearization-based scheduling and the proposed risk-aware control problem by addressing both the risk-neutral and risk-sensitive approaches. Section IV shows the design of a MFTC. Section V presents the GS-MFTC by combining Sections III and IV. Section VI presents the implementation of the GS-MFTC in the non-linear CSTR and shows some numerical results. Finally, some concluding remarks are drawn in Section VII.

II. STOCHASTIC CONTINUOUS STIRRED TANK REACTOR MODEL
Throughout this paper, even though the proposed risk-aware control technique is applicable to other non-lineal systems, we use a stochastic CSTR as the main case study to illustrate both the design and the performance of the GS-MFTC controller. Then, let us consider a stochastic version of the CSTR model presented in [27] (see Figure 1), i.e., The variables of model in (1) are described in Table 2. In Lemma 1 below, we present a detailed deduction of the dimensionless stochastic version of the non-linear model as in [27] by applying Itô's formula.

Lemma 1 (Stochastic Non-linear Model):
The Stochastic CSTR Model in (1) can be expressed as follows: where are the respective dimensionless variables.
Proof: See Appendix.

III. LINEARIZATION-BASED SCHEDULING AND RISK-AWARE CONTROL PROBLEM
We present the scheduling method based on Jacobian linearization of the non-linear system dynamics, i.e., in this case the Stochastic CSTR Model presented in Section II. Let = {1, . . . , n} denote the set of possible operating points, where θ ∈ corresponds to the current operating point given by x θ ∈ R x . Consider a general stochastic non-linear system given by where x ∈ R x and u ∈ R u denote the system states and control inputs, respectively; and E[x] ∈ R x and E[u] ∈ R u denote their expectation values, respectively. Hence, f : is twice differentiable and corresponds to the drift, and c ∈ R to a constant diffusion, where B ∈ R x denotes a standard Brownian motion. The objective is to stabilize the non-linear system in (3) around an operating point , be the error terms for system states and control inputs. Then, the cost functional is given by where Q, Q +Q 0, and R, R +R 0 are symmetric matrices, and the corresponding optimization problem for the non-linear mean-field-type controller is as follows: subject to (3). (5b)

Lemma 2 (Risk Minimization):
The expected value of the cost functional, i.e., EJ (x, E[x],ũ, E[ũ]) , involves variance terms for both the system states and control inputs, i.e., Proof: The proof directly follows after applying the orthogonal decomposition given by ( Let us consider the following linear system dynamics computed from (3): where the state-space matrices are obtained from a Jacobian linearization given by , (7b) Once the stochastic model in (6) is obtained, then a Linear-Quadratic MFTC can be designed to operate around the operating point x θ as discussed in the next section. Thus, the appropriate gains are assigned to the feedback state and its expectation conveniently as the operating points vary along the time (see Figure 2).

IV. LINEAR-QUADRATIC MEAN-FIELD-TYPE CONTROL
We consider two different approaches for the risk-aware control design based on mean-field-type theory. First, we show the risk-neutral approach, which involves the first and second moment terms, i.e., the mean and variance of the system state and the control inputs. On the second hand, we consider the risk-sensitive approach, which involves also higher order terms.

A. RISK-NEUTRAL MEAN-FIELD-TYPE CONTROL
Let us re-write the cost function in (4) by applying Lemma 2 as follows: Thus, the linearized problem corresponding to (5) become a Linear-Quadratic MFTC problem as follows: subject to The solution for the latter mentioned problem is provided in Proposition 1 below.

Proposition 1 (Finite-Horizon Risk-Neutral):
The feedback optimal control inputs of the Risk-Neutral MFTC and the optimal cost, which solve the Problem in (8), are given by where P,P, and δ solve the following ordinary differential equations: with boundary conditions P(T ) = Q(T ),P(T ) = Q(T ) + Q(T ), and δ(T ) = 0.
Proof: See Appendix. Notice that the risk-aware result presented in Proposition 1 is more general than the already well studied and reported LQR and LQG controllers as shown next in Corollary 1.
We present the risk-sensitive MFTC problem, which incorporates higher moments beyond the mean and variance terms [32,Remark 2] in the following section.

B. RISK-SENTITIVE MEAN-FIELD-TYPE CONTROL
By considering the same dynamics linearization, we establish the following risk-sensitive problem: subject to (8b) and (8c). The solution of this problem is presented next in a semi-explicit form in Proposition 2.

Proposition 2 (Finite-Horizon Risk-Sensitive):
The feedback optimal control inputs of the Risk-Sensitive MFTC and the optimal cost, which solve the Problem in (11), are given by: where P,P, and δ solve the following ordinary differential equations: Once the fixed time-horizon MFTC problems are solved, then we present the proposed GS-MFTC approach in the following section.

V. GAIN-SCHEDULED MEAN-FIELD-TYPE CONTROL
In order to design the Risk-Neutral and Risk-Sensitive Gain-Scheduled Mean-Field-Type Control, it is necessary to identify the appropriate gains K θ ,K θ , θ ∈ . The diagram of the GS-MFTC is presented in Figure 2 with n operating points.

A. DESIGN
In order to design the Risk-Neutral GS-MFTC, we solve the infinite horizon Risk-Neutral Mean-Field-Type Control whose respective cost functional J ∞ (x, E[x],ũ, E[ũ]) is given by and the corresponding optimization problem is given by subject to The solution of the infinite horizon problem is presented in Proposition 3 below.

Proposition 3 (Infinite-Horizon Risk-Neutral):
The feedback optimal control inputs of the MFTC and the optimal cost, which solve the Problem in (13), are given by: where the following equations: are solved by P ∞ andP ∞ .
Proof: This result is obtained directly by solving the steady-state equilibrium of the Riccati equations presented in Proposition 1.
The result presented in Proposition 3 shows a control structure involving two main gains, i.e.,K θ and K θ . The risk-aware control scheme is the one presented Figure 2.
On the other hand, since the risk-aware approach is more general than the LQR and LQG counterparts, the LQG control scheme can be directly retrieved from Proposition 3 as shown in Corollary 2 next.

Corollary 2 (Infinite-Horizon Risk-Free):
The feedback optimal control inputs of the Stochastic Mean-Field-Free Control (LQG) and the optimal cost, which solve the Problem in (13) by consideringĀ θ =B θ = 0 andQ =R = 0, are given by where the following equation: Proof: This result is obtained directly by solving the steady-state equilibrium of the Riccati equations presented in Corollary 1.
Differences between the risk-aware and the LQG controller are revealed by the dropping of the variance consideration. Figure 3 presents a less sophisticated feedback control law with respect to the one in Figure 2.
On the other hand, we also solve the Infinite-Horizon Risk-Sensitive Problem, i.e., subject to (13b) and (13c). The solution of the infinite horizon problem is presented in Proposition 4 below.

Proposition 4 (Infinite-Horizon Risk-Sensitive):
The feedback optimal control inputs of the MFTC and the optimal cost, which solve the Problem in (16), are given by: VOLUME 8, 2020 where the following equations: Proof: This result is obtained directly by solving the steady-state equilibrium of the Riccati equations presented in Proposition 2.

B. LOCAL STABILITY OF THE OPERATING POINTS
We analyze the closed-loop stability of the GS-MFTC around the corresponding operating pointsx θ ,ũ θ , for all θ ∈ . Thus, let us consider the following transformation of the state-space model in (13b): Therefore, replacing the optimal control input from either Proposition 3 or Proposition 4 yields Thus, the stability of the operating point is determined by the matrix A θ z . The respective operating point associated with θ ∈ is locally stable if the real component of the spectrum of A θ z is negative, i.e., R{λ(A θ z )} i < 0, for all components i ∈ {1, . . . , 2 x }.

VI. RISK-AWARE NUMERICAL ILLUSTRATIVE EXAMPLE
Let us consider the system parameters presented in Table 3, which have been chosen inspired by the case study presented in [33]. Besides, we consider the following three operating points, i.e., [C θ A T θ R ] where θ ∈ = {1, 2, 3}: corresponding to the following operating dimensionless system states x θ and control inputs u θ , θ ∈ = {1, 2, 3}: = 0.2505. The parameters for the cost functional are Q = 50I 2 ,Q = 80I 2 , R = 1,R = 2. Therefore, the state-space matrices A θ ,Ā θ , B θ ,B θ , σ θ , θ ∈ , corresponding to the operating points are computed from (2) and (7), i.e., where σ 1 = 0.2, σ 2 = 2, and ψ(x 2 ) = exp x 2 γ γ +x 2 . Now, by using the operating points and by applying the result in Proposition 3, the MFTC gains K θ ,K θ , for all θ ∈ , are computed. Thus, it is obtained that   . In order to verify the local stability of the operating points, we compute the matrices A θ z , and the corresponding spectrum λ(A θ z ), for all θ ∈ as follows: showing the local stability of the considered operating points with the designed GS-MFTC. Next, we present the performance of the GS-MFTC over the non-linear model  in (2), and tracking a reference signals C ref A , T ref R changing its value among the different considered operating points C θ A , T θ R , for all θ ∈ according to (18) (see Figure 2) as VOLUME 8, 2020 follows: T ref Figure 4 shows the noise perturbing the stochastic CSTR. The evolution of the reactant concentration is presented in Figure  5. It can be observed that, the reactant concentration tracks the desired reference established in (19). Similarly, Figure  6 shows the evolution of the reactor temperature according to the desired references presented in (20), and Figure 7 shows the evolution of the optimal control input u * and its expectation E[u * ].

VII. CONCLUSIONS
We have studied the design of a Gain-Scheduled Mean-Field-Type Controller (GS-MFTC), which is a risk-aware technique that considers within the system dynamics and cost functional both the mean and the variance of the variables of interest, i.e., system states and control inputs. To this end, we have identified the respective gains for the controller by solving the infinite horizon problem. We have presented a numerical simulation that illustrates the stabilization of the operating points for the non-linear and stochastic Continuous Stirred Tank Reactor (CSTR).
As further work, we propose to study Gain-Scheduled Mean-Field-Type Games (GS-MFTG) under multiple scenarios, e.g., non-cooperative, fully-cooperative, co-opetitive, among others; and implement the GS-MFTG to multiple CSTRs. Finally, imperfect information of the system state is proposed for future investigation.
The deterministic version of the model is presented in [27]. Different from [27], here we present a detail deduction of the corresponding stochastic model using the Itô's formula. Notice that, Thus, The model in (1) is re-written as follows: We apply Itô's formula to the function x 1 , i.e., adding and subtracting D a exp ((x 2 γ )/(γ + x 2 )) yields Similarly, we apply Itô's formula to the function x 2 , i.e., adding and subtracting h 0 A 0 C P F , and, C P T f D a exp x 2 γ γ +x 2 , it finally yields completing the deduction of the non-linear dimensionless model.

PROOF OF PROPOSITION 1
We develop this proof by following the same reasoning as in [14]. Consider the following guess functional: Then, the announced results are obtained by minimizing terms in the latter expression completing the proof.

PROOF OF PROPOSITION 2
This proof is developed by following the same direct method. Thus, the following term appears in the Itôs formula: 2λ Thus, the announced modification over the dynamics P is obtained by performing the match in the identification process completing the proof.