Probabilistic Multi-Step Identification With Implicit State Estimation for Stochastic MPC

Stochastic Model Predictive Control (SMPC) is a promising solution for controlling multivariable systems in the presence of uncertainty. However, a core challenge lies in obtaining a probabilistic system model. Recently, multi-step system identification has been proposed as a solution. Multi-step models simultaneously predict a finite sequence of future states, which traditionally involves recursive evaluation of a state-space model. Particularly in the stochastic context, the recursive evaluation of identified state-space models has several drawbacks, making multi-step models an appealing choice. As a main novelty of this work, we propose a probabilistic multi-step identification method for a linear system with noisy state measurements and unknown process and measurement noise covariances. We show that, in expectation, evaluating the identified multi-step model is equivalent to estimating the initial state distribution and subsequently propagating this distribution using the known system dynamics. Therefore, using only recorded data of an unknown linear system, our proposed method yields a probabilistic multi-step model, including the state estimation task, that can be directly used for SMPC. As an additional novelty, our proposed SMPC formulation considers parametric uncertainties of the identified multi-step model. We demonstrate our method in two simulation studies, showcasing its effectiveness even for a nonlinear system with output feedback.


I. INTRODUCTION
Model predictive control (MPC) is a popular strategy to control multivariable systems with constraints [1].At its core, MPC uses a system model to predict and optimize the future behavior of the system.Obtaining a suitable system model is therefore a central requirement for MPC and has been recognized as a major challenge for decades [2].Traditionally, state-space representations of the prediction model have been used for MPC and other control schemes.A state-space representation is the obvious choice for models that are obtained from first principles and typically employed for data-based system identification [3], [4].Under ideal conditions, state-space models can exactly describe the system dynamics with the fewest number of parameters.To obtain predictions over multiple timesteps, as required The associate editor coordinating the review of this manuscript and approving it for publication was Ton Duc Do .
for MPC, the state-space model can be evaluated recursively.However, a recent trend in system identification is to directly identify multi-step prediction models [5], [6], [7], [8].
A multi-step model can simultaneously predict finite sequences of future states with an individual function for each step of the sequence.At first glance, a multi-step model appears to add unnecessary complexity to the system identification and control task.However, recent work has shown that multi-step identification can have significant advantages over state-space identification [7], [8].Most importantly, multi-step models can have a better accuracy than state-space models and complexity is added only to the offline identification task, not to the online control task.
A closely related trend to multi-step model identification is data-enabled predictive control (DeePC) [9], [10], [11], [12].DeePC draws from behavioral systems theory and combines an implicit multi-step model identification and control task in a single optimization problem.The close relationship between DeePC and MPC with multi-step prediction models has recently been shown in [12], [13].
Another major challenge for the application of MPC is the presence of uncertainty.While nominal MPC provides a certain robustness, it is often not sufficient, especially if the system operates close to safety-critical constraints.A promising approach to handle this situation is stochastic MPC (SMPC) [14], [15].SMPC can be applied when the uncertainty in the model follows a known probability distribution function and it relies on the formulation of chance constraints, that is, constraints with specified probability of violation.
Tackling the challenges of data-based identification and control under uncertainty with state-space and multi-step models is a prominent field of research [6], [7], [8], [17].There are, however, important limitations in previous works using multi-step identification and SMPC.For example, it was previously assumed that the process and measurement noise covariances are known for the identification of the probabilistic multi-step model [8].Unfortunately, this assumption significantly limits the applicability in practice, as the process noise typically represents unknown disturbances of the system.Additionally, most stochastic MPC formulations require state-feedback to solve the optimal control problem [14], [15], which is often not available in practice.
As a main contribution of this work, we propose a stochastic MPC formulation based on multi-step identification without knowledge of the process and measurement noise covariances.Our proposed method is derived for systems with noisy state measurements and we formulate the objective function and chance constraints of the SMPC problem in terms of these measurements.We show that with maximum likelihood estimation, we identify a multi-step model that, in expectation, describes the true distribution of the future measurements of the system.Crucially, this true distribution also considers the uncertainty for the initial state which is estimated from the initial measurement using a Kalman filter [1].Therefore, evaluating our identified multi-step model with noisy state measurements is equivalent to estimating the initial state distribution and subsequently propagating this distribution using the known system dynamics.Finally, we show that this property is a unique advantage of multistep models and does not apply to recursively evaluated state-space models.
The identified multi-step model is thus readily applicable for the formulation of an SMPC problem with noisy statefeedback.As another contribution of this work, our proposed SMPC formulation directly considers the parametric uncertainties of the identified multi-step model.This is in contrast to previous work, where the parametric uncertainties were included as an ellipsoidal uncertainty set [8].As a final contribution, we investigate the proposed SMPC controller based on multi-step models, comparing it to a variant relying on identified state-space models.We conduct this evaluation in two simulation studies, showcasing the effectiveness of SMPC with identified multi-step model, even for a nonlinear system with output feedback.
This paper is structured as follows.In Section II, we introduce the stochastic optimal control problem for a linear dynamic system with known system matrices and noise covariances.In Section III, we assume that this information is not available and discuss probabilistic multi-step identification from data.In Section IV, the proposed SMPC problem based on the identified multi-step model is presented.We investigate our method with a linear building system in Section VI, and a nonlinear CSTR system in Section VII.The work is concluded in Section VIII.

II. STOCHASTIC MODEL PREDICTIVE CONTROL
We consider a linear and uncertain dynamic system given in the discrete-time formulation as: with states x ∈ R n x , inputs u ∈ R n u , measurements y ∈ R n y , process noise e x,k ∈ R n e,x and measurement noise e y,k ∈ R n e,y .The system is described with the matrices x and E y ∈ R n y ×n e,y .As in previous related work [8], we introduce the following assumption.
Assumption 1: The system ( 1) is subject to noisy statefeedback, that is C = I.For practical applications, this assumption can be relaxed to output-feedback, as we demonstrate in Section VII.Furthermore, we assume the following properties of the system noise.
Assumption 2: The system (1) is subject to additive Gaussian process noise and measurement noise, that is: e x,k ∼ N (0, x ) and e y,k ∼ N (0, y ), ∀k.As a main premise of this work, we consider the system matrices A, B, E x and E y and the covariance matrices x and y as unknown.In the following subsection, we derive multi-step models from the state-space representation in (1), and propose an identification method of the parameters of the multi-step model in Section III.

A. MULTI-STEP PREDICTIONS
To formulate the stochastic model predictive control problem in Subsection II-C, we require the probability distribution of the future sequence of system measurements.If the system matrices are known, a multi-step prediction can be obtained by recursively evaluating the state state-space model in (1).Considering Assumption 1, we have: where y ⊤ [1,N ] = y ⊤ 1 , . . ., y ⊤ N denotes a finite sequence of outputs, ⊗ is the Kronecker product and I N is the identity matrix of dimension N .The element in block i, j of the matrices T (•, •) and O(•) can be obtained as: We denote with I [1,N ] the set of integers from 1 to N .Considering (2) and Assumption 2, it follows for the distributed multi-step predictions: with: The covariance in (4c) is obtained by considering the distributions e x,[0,N −1] ∼ N (0, I N ⊗ x ) and e y, [1,N ] ∼ N (0, I N ⊗ y ), the properties of a linear transformation of a normally distributed variable [18, 20.23 b], and the mixed product rule for the Kronecker product [18, 11.11 a].

B. INCORPORATING THE STATE ESTIMATION
The predictive distribution in ( 4) is conditional on the initial state x 0 .However, due to the presence of measurement noise, the initial state x 0 is unknown and must be estimated.The optimal state estimator for system (1) and considering Assumption 2 is the Kalman filter [1].The Kalman filter yields a distribution for the estimated initial state which should also be considered for the distributed multi-step prediction.
In the following lemma, we incorporate the estimated distribution of the initial state to obtain the distribution p(ŷ [1,N ] |y 0 , u [0,N −1] ).In contrast to (4), this distribution is conditional on the noise affected measurement y 0 instead of the unknown x 0 .Obtaining this distribution allows to directly state the stochastic optimal control problem for the measured initial state in the following subsection.Furthermore, we will revisit this distribution for the multi-step identification in Section III.
For the state estimation, we require a prior distribution of x 0 .Without loss of generality and for ease of notation, we assume that this prior has a zero mean.
Assumption 3: The states of the dynamic system are distributed according to x 0 ∼ N (0, x,0 ).This allows to state the following lemma.Lemma 1: Assumption 1-3 hold.The distribution of the multi-step prediction, conditional on y 0 and u [0,N −1] , can be described with: [1,N ] ). ( The mean and covariance in (5) are: where y, [1,N ] is defined in (4c), and the Kalman gain can be computed as: Proof: With observed measurements y 0 , and prior distribution of the states x 0 from Assumption 3, we can perform a Kalman filter correction step [1]: with prior mean x0 = 0, posterior mean x+ 0 and posterior covariance + x,0 .With C = I, due to Assumption 1, and x0 = 0, due to Assumption 3, we obtain: With mean x+ 0 and covariance + x,0 , we have the posterior distribution: Substituting the distribution ( 7) in ( 2) yields (6).□ We omit the prediction step of the Kalman filter in Lemma 1 for reasons that will be clarified in Subsection IV-A.For the discussion in this section, the prediction step is implicitly considered through Assumption 3, that is, we assume to have the true prior distribution of the states.

C. STOCHASTIC OPTIMAL CONTROL PROBLEM
With the distributed multi-step system response (5), we can now state the stochastic optimal control problem.As is commonly the case for data-based MPC [9], we formulate cost and constraints for the measured outputs of the system.Furthermore, we directly incorporate the state estimation in the formulation and consider the uncertainty of the distributed initial state as shown in the previous subsection.
With expectation E(•), and probability P(•), we have the stochastic optimal control problem: with [1,N ] ) obtained from ( 5), positive definite weighting matrices Q ≻ 0, R ≻ 0 and convex constraint set A. We denote the norm ∥y∥ 2 Q = y ⊤ Qy.The n c chance constraints, can be violated with probability ϵ ∈]0, 1[, and are introduced as individual halfspaces [19, 2.2.1] with 117020 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Problem ( 9) is a quadratic optimization problem depending on the measured initial state y 0 and implicitly considers a Kalman filter for the state estimation.

III. PROBABILISTIC MULTI-STEP IDENTIFICATION
For the formulation of the stochastic optimal control problem (9), we require the multi-step distribution in (5).Commonly, the parameters of this distribution are obtained from the system matrices in (1) and the covariances of process and measurement noise from Assumption 2. However, In many practical applications this information is unavailable.
In this section, we propose to use maximum likelihood estimation to identify directly the parameters of the multistep distribution in (5) from system data.In contrast to previous work [8], we do not assume knowledge of the process and measurement noise covariances.As a main contribution, we present Theorem 1.The theorem establishes that, with the expected values for the estimated parameters and covariance matrix, we obtain the distributed multi-step prediction in (5).The identified multi-step model thus implicitly estimates the initial state distribution from noisy measurements.

A. PRELIMINARIES
To simplify the successive notation, we formulate the multi-step prediction model (2) as: We have n z = n x + Nn u independent variables and n t = Nn y response variables.The transposed parameters are and we have additive noise e t ∼ N (0, t ), with t = y, [1,N ] .It follows directly for the distribution of the response variables: Due to measurement noise, the true value of z is unknown and we introduce with e v ∼ N (0, v ) and v = diag(E y y E ⊤ y , 0).We denote with diag(A, B) the block-diagonal matrix with A and B on the diagonal.

B. PARAMETER AND COVARIANCE ESTIMATION
We seek to identify the multi-step system response from data.To this end, we gather m sequences of the dynamic system and introduce: We have the set D = {V , T } with design matrix [21] (10) is compactly evaluated for all samples with: where the vectorized residual matrix E t has the distribution vec(E t ) ∼ N (0, t ⊗ I m ).We require the following assumption for the design matrix.Assumption 4: We have m > n z samples and the design matrix V ∈ R m×n z has full rank, that is, rank(V ) = n z .In practice, Assumption 4 also implies that the gathered input sequences must be persistently exciting [10].
This allows to state the following theorem for the identification of the probabilistic multi-step model.
Theorem 1: Assumption 1 and 2 hold.We have data samples from ( 14) that satisfy Assumption 3 and 4. The bias corrected maximum likelihood approach yields the estimated parameters and covariance matrix: with: We partition the estimated parameters as and have ˆ * t = ˆ * y, [1,N ] .The estimated parameters and covariance matrix have the property: In expectation, the estimated parameters and covariance matrix are thus identical to the true parameters and covariance matrix in (5).Proof: We consider the multi-step model in the form of (12) and have noise disturbed independent variables from (13).Only as an intermediate step of the derivation, we introduce the assumption that u [0,N −1] ∼ N (0, u,0 ).This allows to state the distribution z ∼ N (0, z ) with z = diag( x,0 , u,0 ).We state the joint distribution of t and v as shown in [22]: and the resulting conditional probability follows directly from the properties of a Gaussian normal distribution: with reliability matrix [23]: Furthermore, we introduce (19) as: Of the independent variables, only y 0 is affected by measurement error.We therefore introduce v = diag(E y E ⊤ , 0), and can further simplify (20): The previously introduced u,0 thus vanishes from the expression.We substitute K = diag(L, I) in (19) and consider the definition of the parameters W from (11).We obtain: With the distribution of the response variables in (21), we have the likelihood: To compute the likelihood, we vectorize (15) and consider the properties of the Kronecker product [18, 11.16]: We then maximize the likelihood: Problem (25) has the explicit solution shown in (16) with bias corrected covariance matrix [21], [23].The inverse in (16c) always exists and we have m > n z in (16b) due to Assumption 4. The estimated parameters and covariance have the property that [21], [23]: Considering the definition of Ŵ and ˆ t in (22), we obtain (17).□ As a main consequence of Theorem 1, we have that the estimated parameters and covariance in (16) are, in expectation, identical to the parameters and covariance of the distribution (5).Therefore, we directly obtain the distributed multi-step predictions that incorporates the state estimation.
Finally, we want to remark that the computational complexity of the estimation boils down to the operations in (16) which are tractable even for larger systems.

C. PARAMETRIC UNCERTAINTY
Theorem 1 allows to efficiently estimate the unknown parameters and the full covariance matrix of the probabilistic multi-step model in (5).While we have the favorable property that the estimated parameters are unbiased, we can, in practice, only consider a finite number of samples for the identification task.Consequently, we must consider the parametric uncertainty of the estimated parameters for which we introduce the following theorem.
Theorem 2: Assumptions 1-4 hold, and we have a non-informative prior distribution for the parameters Ŵ .We have m samples of data from (14) and maximum likelihood estimates of the parameter and covariance matrix from (16).Considering the posterior distribution of the parameters, i.e. the parametric uncertainty, we have the distributed multi-step prediction: with: ]. Proof: We consider the multi-step model in the form of (12) and have noise disturbed independent variables from (13).We have estimated parameters Ŵ * and ˆ * t from Theorem 1.The posterior of the identified parameters is For the likelihood, we again consider the vectorized formulation in (24).With the non-informative prior, i.e. p( Ŵ ) ∼ 1, we obtain the posterior distribution for the vectorized parameters vec( Ŵ * ): We then vectorize the linear model (10) for a single test point, yielding: 117022 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Considering the distribution of the vectorized parameters in (28), we obtain a multivariate normal distribution for the response variables with mean E[t] = Ŵ ⊤ v and covariance The first equality stems from the properties of a linear transformation of a normally distributed random variable [18, 20.23 b], and the second equality follows from the mixed product property of the Kronecker product [18, 11.11].For a single test point v, the term v ⊤ ˆ * p v is a scalar and the Kronecker product simplifies to a scalar multiplication: With mean and covariance, we thus have the distribution: From the definition of v, t and considering ( 16), we obtain the distributed multi-step prediction in (26).□ The distributed multi-step prediction including the parametric uncertainty is thus given in (26).We observe the following behavior as the number of samples m used for the identification increases.As shown in [21, 3.3.2],we have for m → ∞ that α(v) → 1.In the same limit, the identified parameters and covariance matrix converge to their expectation in (17) and we recover exactly the distribution (5).We demonstrate this behavior in the numerical example in Section VI.

IV. STOCHASTIC MPC WITH IDENTIFIED MULTI-STEP MODEL
With the distribution in (26) we have the data-based equivalent to (5) including the parametric uncertainty from the identification task.The deterministic formulation of the stochastic optimal control problem (9) with identified multi-step model is given as: The weight matrices Q and R, constraint set A and the chance constraints, defined with a j , b j ∀j ∈ I [1,n c ] and ϵ, are analogous to (9).As a main difference between ( 9) and (30), the mean ŷ * [1,N ] and covariance ˆ * y, [1,N ] now stem from distribution (26).The parametric uncertainty of the identified model results in the factor α(v), introduced in (26c), where ]. Due to the parametric uncertainty, and in contrast to (9), the optimization problem in (30) is not a quadratic problem anymore.We show in Appendix A that problem (30) constitutes a convex second-order cone program [19].In contrast to previous work [8], where the parametric uncertainty is included as an ellipsoidal uncertainty set, we propose to solve problem (30) directly.

A. RECURSIVE APPLICATION
The stochastic optimal control problem in (30) is solved repeatedly at each sampling time to yield closed-loop control actions.This introduces the well known challenges to guarantee recursive feasibility and stability [14].Additionally, we have to consider a limitation of the identified multistep model.As shown in Theorem 1, the multi-step model identifies the parameters and covariance matrix for the distribution in (5).This distribution incorporates a Kalman filter correction step, considering the prior distribution of the initial state p prior (x 0 ) and the current measurement.However, this prior for the initial state distribution is not explicitly formulated.Instead, it coincides with the distribution of the initial state from the sampled data p samp.(x 0 ), as required for Theorem 1.In other words, the multi-step model always considers the same prior distribution, that is, p prior (x 0 ) = p samp.(x 0 ).This has two important limitations.First, it is imperative that the multi-step model is evaluated only for initial states x 0,eval that are reasonably probable in terms of the distribution of the initial state from the sampled data.This is an intuitive limitation, as it is detrimental to perform a Kalman filter correction step with poorly selected prior state distribution.It is, however, a limitation in the sense that data for the identification task must be sampled in the same range that is expected for the closed-loop operation.
The second limitation arises from the fact that with identified multi-step model the prior state distribution is not updated.Updating the prior distribution is typically an important step of Kalman filtering and allows for convergence to the true state distribution over multiple iterations.In practice, this limitation increases the uncertainty of the predicted future measurements and may lead to more conservative control actions of the SMPC controller.

V. STATE-SPACE IDENTIFICATION
To have a comparative baseline for our proposed method, we also employ a probabilistic state-space identification approach.We consider the same data for the identification task which stems from system (1) with unknown parameters and process and measurement noise covariances.In particular, we have the data with individual samples: (31)

A. CHALLENGES WITH STATE-SPACE IDENTIFICATION
We can identify a state-space model as a special case of the described multi-step identification approach with N = 1.
In that case, it follows from (3) that O(A) = A and T (A, B) = B. Unfortunately, state-space identification poses two major challenges which both stem from the requirement that the identified model must be recursively evaluated to obtain the multi-step prediction.
The first challenge is related to the measurement noise of the system.From Theorem 1, we obtain parameters Â * , B * and ˆ * y,1 with the properties: In expectation, the identified parameters are thus identical to the parameters in (6).With: we thus have the conditional distribution ( 5) for N = 1: We can recursively evaluate (34) to obtain a distributed multi-step prediction ŷ[1,N] .Unfortunately, this will not yield the correct multi-step prediction in (5) for N > 1.This is due to the fact that (32) contains contributions from both the process noise and the measurement noise.The recursive evaluation of (34) thus propagates the measurement noise which can significantly inflate the uncertainty of the multistep prediction.
The second challenge with state-space identification is the parametric uncertainty discussed in Subsection III-C.This uncertainty can be considered for a single-step prediction with (26) and N = 1.However, there is no closed-form solution for the recursive application of (34) if parametric uncertainties are considered.

B. IDENTIFIED STATE-SPACE MODEL FOR SMPC
The distributed multi-step system response obtained from the recursively evaluated state-space model is expected to have significant shortcomings due to the challenges discussed above.Regardless, we suggest an approach to formulate the stochastic MPC problem with the identified state-space model.The approach serves as an important comparative baseline, as it is obtained under the same premises as for the multi-step identification.
We formulate the stochastic MPC problem as in (9), using the identified state-space model with parameters Â * , B * and covariance ˆ * y,1 .The distributed multi-step prediction is obtained by recursively evaluating (34) without considering the parametric uncertainty.identical to their expectation in (17), and we recover the true distribution in (5) from data.On the other hand, MSM b is identified with only m = 100 samples and therefore experiences significant parametric uncertainty, according to Theorem 2. However, while the obtained distribution is broader and the mean is shifted, it still approximates the true distribution in (5).

C. STOCHASTIC MPC
We formulate a stochastic MPC controller based on the identified multi-step model (MSM-SMPC).As a comparative baseline, we also formulate a stochastic MPC controller based on an identified state-space model (SSM-SMPC).Both models are obtained from the same data as described in the previous subsection.
For the control objective, we choose the cost function: The actuators (heating and cooling power) are limited to −6 kW ≤ Qi ≤ 6 kW for all rooms.We have the individual chance constraints P(T i ≥ 18) ≥ (1 − ϵ), which can be violated with probability ϵ = 10 −3 .The stochastic optimal control problem is implemented and solved using CasADi [26] with IPOPT [27].
In a first investigation, we compare the open-loop predictions of both controllers.We define a scenario with initial state x 0 = [23,20,20,20, 10] ⊤• C and corresponding noise disturbed measurement y 0 .We then solve the optimal control problem in (30) for the multi-step model and ( 9) for the state-space model.As discussed in Section V, the SMPC controller based on the state-space model does not consider the parametric uncertainty.The resulting open-loop predictions are shown in Figure 2. We display the predicted mean ŷ[1,N] and standard deviation, considering c p (ϵ) for the chosen violation probability.Furthermore, we show 50 samples of the true system response for the optimal open-loop input sequence, considering the same initial state and randomly drawn process and measurement noise.It can be seen in Figure 2 that the MSM-SMPC controller yields a suitable sequence of inputs to minimize the cost function while satisfying the chance constraints.While the chance constraints are also satisfied for the SSM-SMPC controller, the predicted uncertainty bounds are much larger and overly conservative.In the investigated scenario, with forecasted ambient temperature of T f = 10 • C, this leads to suboptimal controls action with unnecessarily high energy consumption.This effect is expected, as discussed in Section V, as the identified state-space model incorrectly propagates the measurement noise.
In a second investigation, we compare the closed-loop performance of the two controllers.To this end, we consider the same initial state as in the previous section and recursively control the system for a period of 50 h with either the MSM-SMPC or SSM-SMPC controller.The samples differ in the randomly drawn sequences of process and measurement noise.Finally, we obtain the percentage of constraint violations and the total energy consumption for each run and compute mean and standard deviation of these values.The performance metrics are shown in Table 1.The MSM-SMPC controller yields on average 8.3 % less energy consumption than the SSM-SMPC controller.Both controllers satisfy the chance constraints with no violations.

VII. NONLINEAR CASE STUDY WITH OUTPUT-FEEDBACK
The proposed multi-step identification approach is derived for linear systems with noisy state-feedback.Additionally, we explore its applicability to a nonlinear system with output-feedback in this section.The application to nonlinear systems is motivated by interpreting the process noise in (1a) as an additive nonlinear term.While this interpretation violates Assumption 2, that is, the process noise is not normally distributed, we expect to identify a multi-step model where the identified uncertainty approximately encompasses the nonlinearities.
As in the previous section, we compare the SMPC based on an identified multi-step model with a variant based on state-space identification.

A. SYSTEM DESCRIPTION
For the nonlinear case study, we consider the continuously stirred tank reactor (CSTR) previously introduced in [28].The reactor is modeled with four states x ⊤ = [c A , c B , T R , T k ], with the concentrations c A [mol L −1 ] and c B [mol L −1 ] and the temperatures T R [ • C] of the reactor and T K [ • C] of the cooling jacket.As control inputs, we have the flow rate V [m 3 s −1 ] which is normalized with the reactor volume V R [m 3 ], yielding F = V /V R , and the heat removed from the jacket Q [kJ h −1 ].For the safe operation of the CSTR, the states and inputs must lie within the bounds shown in Table 2.In our investigated scenario, the nonlinear TABLE 2. Bounds for the CSTR system. 1 Chance constraint for SMPC.
The nonlinear model is created and simulated in dompc [29] with a timestep of 18 s.The model equations and parameters can be found in [28] or online. 2

B. SYSTEM IDENTIFICATION
As in the previous section, we compare the identified state-space model (SSM) and multi-step model (MSM).Furthermore, we also investigate the effect of having full state-feedback vs. output-feedback for system identification and the successive control application.As in related databased approaches [5], [6], [9], [11], output-feedback can be incorporated in the proposed methods by introducing the state: If the system is observable, and with l chosen larger than the system lag, the introduction of state (36) allows to reformulate the system with output-feedback in the form of (1) [11] and satisfies Assumption 1.Unfortunately, considering output-feedback, by introducing the state in (36), violates Assumption 2 as the additive measurement noise on the newly introduced state (36) is now correlated.While this correlation is known to yield biased estimates [3, Sec.5], we show in the following example that the proposed method can still lead to good results.
For the system identification, we gather m = 500 simulated sequences of length L = N + l with N = 20 and l = 1 for the models with state-feedback and l = 3 for model with output-feedback.For both cases we also create m = 100 sequences of test data.All sampled sequences are created with uniformly random initial state, within the bounds shown in Table 2, and with persistently exciting random inputs.
In contrast to the linear case study in Section VI, there is no ground-truth distribution to evaluate the performance of the identified models.We therefore investigate the quality of the obtained probabilistic models by computing the logarithm of the predicted distribution, that is: for the test samples y The logpredictive density is an expressive measure for the quality of the identified probabilistic model.High values indicate high confidence in correct predictions, while low values indicate high confidence in incorrect predictions.
We display the log-predictive density in Figure 3 in the form of a box-plot to visualize the median, the quartile range, minimum and maximum as well as outliers.The identified multi-step model shows clear advantages over the recursively evaluated state-space model.Both, for state-feedback and output-feedback, the median and quartile ranges of the log-predictive density are significantly higher for the multi-step model.We also see that the log-predictive density obtained with the MSM is increased in the case of  state-feedback.The state-space model, on the other hand, experiences more severe outliers for state-feedback, reducing the overall log-predictive density.We reason that with state-feedback the uncertainty in the predictive distribution is reduced as more information is available.However, this only benefits the log-predictive density of the MSM which yields accurate predictions with high confidence in that case.

C. STOCHASTIC MPC
We proceed our investigation with an analysis of the stochastic MPC controllers obtained with the multi-step model (MSM-SMPC) and the state-space model (SSM-SMPC).Furthermore, we continue to investigate the differences between output-feedback and state-feedback for the identified systems and the control application.
All investigated controller variants are implemented with the control objective to maximize the production of component B. Furthermore, we seek to penalize rapid changes of the control inputs.To this end, we propose the cost function: The problem is formulated with horizon N = 20 and must consider the constraints in Table 2. Of those constraints, we formulate the safety critical bound of the reactor temperature as a chance constraint P(T R ≤ 135) ≥ (1 − ϵ), with probability of violation ϵ = 10 −3 .
To illustrate the behavior of the MSM-SMPC and SSM-SMPC controllers, we showcase an exemplary closedloop trajectory in Figure 4, for the case of output-feedback.Both controllers run for 40 timesteps, corresponding to 12 min of simulation time.Additionally, we display mean and standard deviation, scaled with c p (ϵ), of the open-loop prediction for the final timestep.Both controllers achieve satisfactory control performance in the example, yielding a high concentration of c B while safely avoiding the constraint T R ≤ 135.However, we see in Figure 4 that the SMPC controller with multi-step model can operate significantly closer to the chance constraint for the reactor temperature.This allows to realize a higher product concentration c B and higher normalized flow rate F, which yields overall more product.
To further quantify the performance of SSM-MPC and MSM-SMPC, also for the case of state-feedback, we present the results in Table 3.We display the amount of produced component B and the maximum constraint violation of the chance constraint T R ≤ 135 for all investigated controller variants.These performance indicators are computed as mean and standard deviation for 50 independent experiments with different initial state sampled uniformly within the bounds in Table 2.The table supports the qualitative finding from Figure 4. MSM-SMPC outperforms SSM-SMPC in the scenario with state-feedback and for the scenario with outputfeedback.For the output-feedback scenario, we also see that SSM-SMPC leads to minor constraint violations.Finally, we observe that both controllers have a better performance, in terms of the obtained product, when state-feedback is available.In the case of state-feedback the MSM-SMPC controllers results, on average, in 16.4 % more product than the SSM-SMPC controller.

VIII. CONCLUSION
In this work, we propose a novel approach that combines probabilistic multi-step system identification with stochastic Model Predictive Control (SMPC).Our identification procedure is derived for linear systems with noisy state-feedback and with Gaussian process and measurement noise.In contrast to previous work, our proposed method does not require knowledge of the noise covariance matrices.
As a main contribution, we derive that the identified multi-step model yields, in expectation, the true distribution of the future measurements.We show that evaluating our identified model with noisy state-measurements is equivalent to estimating the initial state distribution and propagating this distribution with the known system dynamics.In this way, the identified multi-step model performs an implicit state estimation and can directly be used to formulate an SMPC problem for noisy state-measurements.
We demonstrate the theoretical findings and the performance of our proposed data-based SMPC controller in two simulation studies.In comparison to a SMPC controller based on an identified state-space model, we achieve significantly better performance and safer operation.Furthermore, we showcase in the second simulation study that our proposed method can also be applied to a nonlinear system with outputfeedback, despite not being originally derived for this context.
In future work, we seek to rigorously extend our method to linear systems with output-feedback, by considering the correlation of the measurement noise.Furthermore, we seek to extend the method by updating the identified multi-step model with new data in a recursive fashion. .

FIGURE 1 .
FIGURE 1.Distributed multi-step prediction of the building system for an initial measurement y 0 and random control sequence u [0,N−1] for N = 12.Comparison of true distribution(5) and two identified multi-step models (MSM).The distribution of MSM a with m = 1000 samples and MSM b with m = 100 samples are obtained according to(26).

FIGURE 2 .
FIGURE 2. Building system: Comparison of open-loop prediction (mean and predicted standard deviation) of SMPC with multi-step model and SMPC with state-space model for the same initial measurement.The standard deviation is scaled with c p introduced in (9).Using the open-loop control inputs, 50 samples of the true system response are drawn.

FIGURE 3 .
FIGURE 3. Nonlinear CSTR system: Comparison of log-predictive density for the identified state-space model (SSM) vs. the multi-step model (MSM).The predictive density is computed for 100 test cases as shown in (37).Representation as box-plot with filtered outliers, inter-quartile range, minimum-maximum range and median.

FIGURE 4 .
FIGURE 4. Nonlinear CSTR: Output-feedback SMPC closed-loop results and future prediction with uncertainty bounds.Comparison of SMPC and prediction with identified multi-step model (MSM) and identified state-space model (SSM) for an exemplary initial condition.

TABLE 1 .
Building system: Closed-loop performance over a period of 50h with mean and standard deviation computed over 20 samples.

TABLE 3 .
Nonlinear CSTR system: Closed-loop SMPC performance over 50 experiments.Comparison of MSM and SSM models with state or output-feedback.