Stochastic Relative Degree and Path-wise Control of Nonlinear Stochastic Systems

We address the path-wise control of systems described by a set of nonlinear stochastic differential equations. For this class of systems, we introduce a notion of stochastic relative degree and a change of coordinates which transforms the dynamics to a stochastic normal form. The normal form is instrumental for the design of a state-feedback control which linearises and makes the dynamics deterministic. We observe that this control is idealistic, i.e. it is not practically implementable because it employs a feedback of the Brownian motion (which is never available) to cancel the noise. Using the idealistic control as a starting point, we introduce a hybrid control architecture which achieves \emph{practical} path-wise control. This hybrid controller uses measurements of the state to perform periodic compensations for the noise contribution to the dynamics. We prove that the hybrid controller retrieves the idealistic performances in the limit as the compensating period approaches zero. We address the problem of asymptotic output tracking, solving it in the idealistic and in the practical framework. We finally validate the theory by means of a numerical example.

This article has been accepted for publication by IEEE Transactions on Automatic Control.

The manuscript included in this file is the open access accepted version.
This open access version is released on arXiv in accordance with the IEEE copyright agreement.
The final version will be available at IEEE Xplore. Abstract-We address the path-wise control of systems described by a set of nonlinear stochastic differential equations. For this class of systems, we introduce a notion of stochastic relative degree and a change of coordinates which transforms the dynamics to a stochastic normal form. The normal form is instrumental for the design of a state-feedback control which linearises and makes the dynamics deterministic. We observe that this control is idealistic, i.e. it is not practically implementable because it employs a feedback of the Brownian motion (which is never available) to cancel the noise. Using the idealistic control as a starting point, we introduce a hybrid control architecture which achieves practical path-wise control. This hybrid controller uses measurements of the state to perform periodic compensations for the noise contribution to the dynamics. We prove that the hybrid controller retrieves the idealistic performances in the limit as the compensating period approaches zero. We address the problem of asymptotic output tracking, solving it in the idealistic and in the practical framework. We finally validate the theory by means of a numerical example.
Index Terms-Stochastic systems, relative degree, normal form, feedback linearisation, output tracking.

I. INTRODUCTION
A point of departure in the study of nonlinear deterministic systems is the definition of the relative degree of the system and, consequently, of a change of coordinates that is able to transform the differential equations in a so-called normal form that makes analysis and control easier. These ideas were first introduced in the seminal work [1], where the authors solved the problem of static state-feedback non-interacting control. The theory of normal forms was later addressed in [2] and [3] for the control and observation of time-varying nonlinear systems, and a systematic overview of normal forms was given in [4]. The problem of feedback linearisation of single-input single-output and multi-input systems was introduced in [5] and in [6], respectively, and a systematic procedure to find the feedback-linearising control was provided in [7] and [8]. In [9] the notion of zero dynamics was introduced and later employed in [10] to tackle the problem of asymptotic stabilisation of nonlinear systems.
In this paper we introduce the notion of stochastic relative degree to develop a theory of path-wise feedback control for a general class of systems described by nonlinear stochastic differential equations. The advantage of using this stochastic framework lies in the fact that it allows to address uncertainties characterised by probabilistic properties which cannot *This work is partially supported by Shanghai University under the "Foreign Expert Program" of Shanghai University (No. 21WZ0109).
Alberto Mellone and Giordano Scarciotti are with the Department of Electrical and Electronic Engineering, Imperial College London, London, SW7 2AZ, UK. {a.mellone18,g.scarciotti}@imperial.ac.uk be captured by the classical deterministic robust designs. Modelling in the stochastic framework is flexible and lends itself to mechanical systems (e.g. the quarter-car model), electro-mechanical systems (e.g. the suspended gyro), aerospace systems (e.g. the satellite dynamics) and mathematical finance. A survey of these applications can be found in [11,Section 1.9] and references therein. Stochastic differential equations are also at the basis of methodological applications such as stochastic H ∞ control (see [12], [13] and [14]), filtering and optimal control (see [15] and [16]).
Some notions of normal forms for stochastic systems have been introduced in the literature. For example, in [17] and [18], Stratonovich calculus was used to obtain a normal form for purely diffusive processes, while [19] employed coordinate changes to introduce symmetries for stochastic differential equations. One of the first works suggesting the convenience of a normal form for control of stochastic systems is [20], where the change of coordinates makes the dynamics quasi-linear. In [21] and [22], similar coordinate transformations are proposed in order to reduce the system dynamics to canonical forms that are amenable to specific control strategies. Specifically, strict-feedback noise-prone dynamics are obtained, which allow achieving optimal globally stabilising back-stepping controllers as proposed in [23].
The differences between the normal forms mentioned above and the one we introduce in this paper are both technical and in scope. From a technical viewpoint, we propose a coordinate projection and a feedback control that annihilate the noise in the linearised coordinates. From an objective viewpoint, the stochastic relative degree and the normal form are mere instrumental means for the design of a practical hybrid controller which achieves output tracking in a path-wise fashion. In other words, the controller that we develop compensates for each specific realisation of the stochastic disturbance.
More generally, by defining a new stochastic normal form, the goal of this paper is to address the path-wise control of stochastic systems described by a general class of nonlinear stochastic differential equations. We show that the implementation of feedback laws that perfectly linearise the system dynamics in a new set of coordinates comes with insurmountable causality issues, hence the attribute idealistic with which we refer to these control laws. In fact, they employ a feedback of the noise, which is not practically available. However, with these idealistic controls at hand, we introduce hybrid nonlinear controllers that incorporate a causal estimator of the Brownian motion. This controller is practically realisable and, although achieving only approximate feedback linearisation and tracking, its performance can be arbitrarily improved by tuning an underlying parameter, retrieving the idealistic case as a limit behaviour.
While preliminary work has been published in [24], [25], [26], in this paper we present several additional contributions. The main novelties introduced are as follows. 1) All the results are now proved, which provides a substantial theoretical contribution. 2) By leveraging the Itô-Stratonovich equivalence, we are able to obtain sharper results in the theory of the stochastic normal form. 3) We provide a characterisation of the solvability of the feedback linearisation problem. 4) The theory of the practical control has been revised and made sharper. 5) An analysis of the control challenges arising when the input appears in the diffusion term of the equation has been added. 6) Practical asymptotic output tracking has been addressed and solved for the first time. 7) A number of technical results regarding uniform asymptotic stability of nonlinear timevarying stochastic systems have been added and proved. For the sake of readability, they have been gathered in the Appendix. 8) A comprehensive example illustrates the path-wise output tracking of a nonlinear stochastic system both in the idealistic and practical scenarios.
The paper is organised as follows. In Section II we recall some preliminary notions on stochastic systems. Section III introduces the stochastic relative degree and normal form. In Section IV we address the problem of feedback linearisation in the idealistic framework. In Section V we propose a hybrid controller, which practically approximates the idealistic linearising control. Section VI addresses the problem of asymptotic output tracking. In Section VII we validate the theory by means of a numerical example. Section VIII contains some concluding remarks. Finally, technical lemmas which are instrumental to prove some of the results of the paper have been collected and proved in the Appendix.
Notation. The symbol Z denotes the set of integer numbers, while R and C denote the fields of real and complex numbers, respectively; by adding the subscript "< 0" ("≥ 0", "0") to any symbol indicating a set of numbers, we denote that subset of numbers with negative (non-negative, zero) real part. Where convenient, the symbol ∂ n x is used as a shorthand for the operator ∂ n /∂x n , while α (n) indicates the n-th time derivative of α. The Lie derivative of the smooth scalar function h(x) along the vector field f (x) is denoted by L f h(x). We use the . Given two smooth vector fields f (x) and g(x), we define the operator ad f g( . (∇, A, P) is a probability space given by the set ∇, the σalgebra A defined on ∇ and the probability measure P on the measurable space (∇, A). A stochastic process with state space R n is a family {x t , t ∈ R} of R n -valued random variables, i.e. for every fixed t ∈ R, x t (·) is an R n -valued random variable and, for every fixed w ∈ ∇, x · (w) is an R n -valued function of time [27,Section 1.8]. For ease of notation, we often indicate a stochastic process {x t , t ∈ R} simply with x t (this is common in the literature, see e.g. [27]). With a slight abuse of notation, any subscript different from the symbol "t" indicates the corresponding component of the vector x t , e.g. x i is the i-th component of the vector x t . All mappings appearing as integrands in stochastic integrals are assumed to be integrable in the corresponding sense, namely Itô's (see, e.g., [16,Definition 3.1.4]) or Stratonovich's (equivalent to deterministic integrability).

II. PRELIMINARIES
In this section we shortly recall the theory of generalised stochastic processes and define differential operators that will be used in the remainder of the paper.
Let C ∞ 0 (R) be the space of all infinitely differentiable functions on R with compact support [28, Definition 1.2.1]. The following definitions characterise the notions of distribution (also known as generalised function), distributional derivative and generalised stochastic process.
Definition 2. [28, Definition 3.1.1] For any distribution ψ, its distributional derivativeψ is defined as the distribution that satisfiesψ Note that generalised functions have derivatives of all order, which are generalised functions as well.
Definition 3. [27, Section 3.2] A generalised stochastic process is a random generalised function in the sense that a random variable ψ(ϕ) is assigned to every ϕ ∈ C ∞ 0 , where ψ is, with probability 1, a generalised function.
We now look at the Brownian motion as a generalised stochastic process. Therefore, its distributional derivative is always defined [27,Section 3.2]. In particular, the generalised stochastic process given by such a derivative has zero mean value and covariance function given by the generalised function δ(t − s), t, s ∈ R, i.e. the Dirac delta. Consequently, the derivative of the generalised Brownian motion is the generalised white noise [27,Section 3.2]. In the remainder, with a slight abuse of notation, we refer to generalised Brownian motion and generalised white noise omitting the attribute "generalised" and we denote them by simply W t and ξ t , respectively, with ξ t =Ẇ t . It should be emphasised that the just mentioned time derivative is meant in the sense of distributions, and not as the limit of the difference quotient as the increment tends to zero, which instead applies to differentiable functions in the classical sense.
Consider the nonlinear single-input, single-output stochastic system expressed in the shorthand integral notation by with x t ∈ R n , u ∈ R, y t ∈ R and f : R n → R n , g : R n → R n , l : R n → R n , m : R n → R n , h : R n → R smooth functions, i.e. they admit continuous partial derivatives of any order. We assume that, for a fixed initial condition x t=0 , the solution of (1) is unique. Note that, in the light of the previous discussion, system (1) can be rewritten in the following differential notatioṅ Note that when ξ t is (generalised) white noise, as in this case, then the differential equation (2) is equivalent to the integral equation (1) if the latter is interpreted in Itô's sense [27,Section 10.3]. Given the equivalence of the two representations in the framework of generalised stochastic processes, in the remainder of the paper equations (1) and (2) are used interchangeably, as convenient, to refer to the same nonlinear stochastic system. We refer the reader to [27,Chapter 3] for a detailed discussion on the relation between Brownian motion and white noise, and the representations (1) and (2).

III. STOCHASTIC RELATIVE DEGREE AND NORMAL FORM
In this section we introduce the concept of stochastic relative degree and show that a suitable coordinate transformation brings the system into a simpler form, which is convenient for analysis and control.
We first introduce three new operators, which are fundamental to systematically define repeated time derivatives of stochastic processes. The first one, which indicates the second derivative of h along the vector fields f and g, is defined as Similarly to the Lie derivative, we use the notation , to indicate the reiterated operations. The second operator l S f h, which we call the stochastic Lie derivative, indicates the derivative of h along the drift vector field f and diffusion vector field l, namely The reiterated application of this operator can be defined if the white noise does not appear explicitly. That is, if is a deterministic expression, we Having defined the operators G , S and A, it is easy to see that, by using Itô's formula, the first derivative of the output of system (2) is given by We now define the concept of stochastic relative degree and then point out the rationale of such a definition.
As it is formally proved in the next proposition, in Definition 4, condition (ND) (which stands for noise decoupling) ensures that the noise ξ t does not appear in y t and its first r − 1 derivatives. In the remainder we will omit the dependency of the operators S and A on the white noise ξ t whenever this does not appear explicitly (for instance because of condition (ND)). Condition (CD) (for control decoupling) ensures that the control input u does not appear in y t and its first r − 1 derivatives. Finally, condition (RD) (for relative degree) ensures that the control u appears in the r-th derivative of y t , thus defining the relative degree of the system. These observations are formally gathered in the following result, the proof of which can be found in Appendix A.
Proposition 1. Suppose that system (2) has stochastic relative degree r > 0 atx. Then i.e. the first r − 1 derivatives of y t do not depend explicitly on ξ t nor u. Moreover, if, at timet, xt =x, then The previous result shows that, analogously to the deterministic case, the stochastic relative degree is equal to the order of the derivative of the output at timet in which the input u(t) explicitly appears. Two observations are in order: first, while the white noise does not appear in all the derivatives up to order r − 1 because of condition (ND), it may or may not appear in the r-th derivative; second, differently from the deterministic case, the control u appears linearly and/or quadratically in (6). Remark 1. Condition (ND) is a type of disturbance decoupling condition, as we suppose that the noise does not appear in y t and its successive r − 1 derivatives. For a deterministic analogous, see e.g., [30,Section 4.6]. If L l l S k f h(x) = 0 for a k < r − 1, the differentiation of y t up to the r-th time would require us to introduce successive derivatives of the white noise, which is theoretically and practically challenging. We exclude this possibility with (ND). Remark 2. The equality L m l S k f h(x) = 0 appears in both conditions (ND) and (CD). This repetition is unnecessary. However, since L m l S k f h(x) multiplies both ξ t and u in the expression of the derivative y (k+1) t , we believe that for the sake of clarity it is beneficial to require it to be zero in both the noise and control decoupling conditions. Remark 3. Consider a linear stochastic system, i.e. system (2) By applying Definition 4, system (2) in the linear case has stochastic relative degree r if Note that the conditions are remarkably simple and reminiscent of the deterministic case. In fact, because the Hessian of a linear function with respect to x is identically zero. Assuming that the relative degree of this system is, for instance, r > 2, it follows that Having defined the notion of stochastic relative degree and having discussed its interpreation, we are now interested in finding a diffeomorphism Φ : R n → R n that locally (i.e. in a neighbourhoodŪ ofx ∈ U ⊂ R n ) transforms system (2) in such a way that its dynamics is somewhat "simpler". The diffeomorphism we are looking for is a direct consequence of the definition of the stochastic relative degree previously given. To simplify the exposition, we make the following assumption on the stochastic Lie derivatives of y t = h(x t ) along the drift vector f and the diffusion vector l. Assumption 1. Let r be the stochastic relative degree of system (2) atx. Then the row vectors

are linearly independent.
Observe that if Assumption 1 holds, then necessarily r ≤ n.
Remark 4. For deterministic nonlinear systems, the linear independence of the gradients of the first r − 1 successive derivatives of the output atx is a consequence of the relative degree being defined, see e.g, [30, Lemma 4.1.1]. In Section III-A we prove that this property holds in the present setting for the case in which m ≡ 0 nearx. However, a proof that a similar result holds in general (m = 0) is missing. For simplicity, we use Assumption 1 at this stage to develop the theory of normal form for the most general class of systems, and we note that no counter-example has been found for which this property is not satisfied when m = 0 nearx. Proposition 2. Suppose that system (2) has stochastic relative degree r atx and let Assumption 1 hold. Set If r < n, then there exist smooth functions φ r+1 (x), ..., φ n (x), with φ j ∈ R for all j ∈ {r + 1, .., n}, such that the Jacobian of the mapping is invertible atx almost surely, thus defining a coordinate transformation in a neighbourhood ofx. Then the state-space where the mappings 1 c, b, p j and q j are affine in ξ t .
Proof. By Assumption 1, the matrix has rank r atx. If r < n, let γ r+1 (x), ..., γ n (x), with γ j ∈ R n for all j ∈ {r + 1, .., n}, be any set of n − r vectors such that has rank n. Note that this is possible because there always exist n − r linearly independent vectors γ r+1 (x), ..., γ n (x) that complete the first r linearly independent vectors (7) is a local diffeomorphism in a neighbourhood ofx and, therefore, it defines a local change of coordinates z t = Φ(x t ) for the stochastic system (2). Applying Itô's lemma and since the system has relative degree r, the following holdṡ Moreover, We now set As for the remaining n − r components of z t , by applying Itô's lemma to the functions φ j (x t ) and setting The proof is completed by observing that y t = h(x t ) = z 1 and that, by the definitions of the operators S and A, the coefficients c, b, p j and q j are affine in ξ t .
Note that it might be possible to find smooth functions φ r+1 , ..., φ n such that the dynamics of the last n−r transformed coordinates is independent of the input u, i.e. q j (·, z t ) ≡ 0, s j (·, z t ) ≡ 0, for all j ∈ {r + 1, ..., n}, in a neighbourhood of Φ(x). This observation motivates the next definition.
Definition 5. (Stochastic Normal Form) Let x t be the unique solution of (2) and z t = Φ(x t ) be a local diffeomorphism in a subset U of R n such thaṫ System (9) is said to be the stochastic normal form of system (2).
Remark 5. The fact that the coefficients c, b, p j and q j are affine in ξ t guarantees that system (8), which is written in the differential notation (as in (2)), can always be equivalently written in the integral notation (as in (1)).
For compactness, in the remainder we use the definitions p = p r+1 . . . p n , q = q r+1 . . . q n and s = s r+1 . . . s n . Obviously, if the stochastic relative degree atx is equal to the order of the system, then the system admits a stochastic normal form in a neighbourhood U ofx (because p, q and s have dimension zero).
Remark 6. For deterministic systems it can be proved (see, e.g., [30,Proposition 4.1.3]) that functions φ r+1 , ..., φ n always exist such that a normal form exists when r < n. While a proof that this property holds in general in the present setting is missing, in Section III-A we prove that this property holds for the case in which m ≡ 0 nearx. Remark 7. The notion of stochastic relative degree and normal form presented are consistent with Itô's interpretation. This is without loss of generality, as all the results of this section can be obtained also in other formalisms, e.g. Stratonovich's formalism [31].

A. Sharper Results for m ≡ 0
In the previous section we introduced the concept of relative degree and of normal form for a class of nonlinear stochastic systems in which the control input appears both in the drift and in the diffusion terms of the stochastic differential equation. This allowed us to provide as general as possible definitions. However, for the remainder of the article, it is beneficial to make the standing assumption that the control u does not enter the diffusion term of the stochastic differential equation (i.e. m ≡ 0) in a neighbourhood ofx. On the one hand, this sub-class of systems is more common in the literature (see e.g. [21], [22] and references therein). On the other hand, this assumption allows us to achieve sharper results in terms of nonlinear control of stochastic systems, both in the idealistic case i.e. when the noise process is assumed available) and, in the practically implementable controller that we develop in Section V-B. A discussion of the general case (i.e. m = 0) is given in Section V-C. Therefore, for the time being, we assume m(x t ) ≡ 0 nearx and we consider systems of the form (10) or, equivalently, In order to prove results in this section it is useful to introduce the Stratonovich equivalent of system (10), given by ∂x l(x), and the symbol • denotes the fact that the stochastic integral is meant in Stratonovich's sense. The Itô solution x t of (10) is identical to the Stratonovich solution of (12) (see, e.g., [32]).
We now discuss what implications follow from restricting ourselves to the class of systems (11). Firstly, the operator m g A l , reduces to L g , the Lie derivative along the only control vector field g. Secondly, as the control input does not appear in the diffusion term, the derivative of any function of the state x t is never quadratic in the control input u. In fact one can observe that u 2 multiplies the operator m G m , e.g. in (4), and that this operator is identically zero nearx because so is the vector field m. Having said this, the definition of stochastic relative degree becomes simpler and for the sake of clarity it is useful to rewrite it.
for all x in a neighborhood ofx and for all k ∈ {0, ..., r − 2}.
By considering a class of systems of the form (11), we can achieve sharper results in terms of definition of a stochastic normal form. Indeed, we can now prove the claim previously assumed in Assumption 1. To this end, we first prove a technical result.
Lemma 1. Let r be the stochastic relative degree of system (11) atx. Then Proof. First observe that by (ND), for all k in {0, ..., r − 2} nearx. Then the claim follows by induction. In fact, (13) trivially holds for k = 0. Note that for some 0 < k < r − 1 we have, in view of (14), that Rewriting this equation using in the Lie derivative notation and assuming by the inductive hypothesis that (13) holds for 0 < k < r − 1 yields The fact that the stochastic Lie derivatives of h along f and l are identical to the Lie derivatives of h along f S up to order r − 1 is crucial to prove the rest of the results in this and the next section. In fact, this equivalence allows us to leverage the deterministic techniques, because the Stratonovich differentiation rule is formally identical to the deterministic one. We can then use the identities in Lemma 1 to translate the results to the Itô system (11).
We are now ready to prove the following.
Lemma 2. Let r be the stochastic relative degree of system (11) atx. Then the row vectors are linearly independent.
Proof. Lemma 1 and the definition of stochastic relative degree of (11) imply that the set of conditions Using again Lemma 1, the claim follows.
For systems of the form (11) the result in Proposition 2 can be specialised to the existence of a coordinate transformation z t = Φ(x t ) such that the transformed dynamics is given bẏ where the coefficients c and p j are as before and Observe that, unlike the general case in which m = 0, since now the coefficients q j are simply the Lie derivatives of φ j along g for j = r + 1, . . . , n, it is easy to show that it is always possible to find φ r+1 , . . . , φ n such that said coefficients are identically zero for x in a neighbourhood ofx. Exploiting Lemma 1, the proof is analogous to the one reported in [30,Proposition 4.1.3]. Thus, we can conclude that it is always possible to find a coordinate transformation Φ(x) in a subset U of R n such that the dynamics in the transformed state z j = p j (ξ t , z t ), j = r + 1, ..., n, which is the normal form of system (11).
Remark 8. The notion of normal form which we propose introduces elements of significant novelty, compared to works on analogous topics like [20], [21] and [22]. By condition (ND) in Definition 6, the state is projected onto new coordinates, of which the first r − 1 have a noise-free dynamics. Not only is this a fundamental difference with respect to past works, but it also allows us to introduce, as shown in Sections IV and V, control laws performing path-wise control of nonlinear stochastic systems. To the best of the authors' knowledge, this problem has not been systematically addressed in the literature.
IV. IDEALISTIC LINEARISATION VIA STATE FEEDBACK We now turn our attention to the problem of exact feedback linearisation. Specifically, in this section we formulate and solve an idealistic version of the problem, by assuming that the control input is a function of the noise ξ t . Although unrealistic because it implies that the noise is available for feedback, this assumption is necessary to develop the theory of exact linearisation via feedback. This theory is instrumental for the introduction of a practically implementable controller in Section V which, by approximating the noise contribution to the dynamics via measurements of the state, achieves a practical result. The feedback linearisation problem is formally defined as follows.

Problem 1. (Exact Feedback Linearisation) Consider the nonlinear stochastic systeṁ
Given a pointx, the problem of exact feedback linearisation consists in finding a neighbourhood U ofx, a feedback law u t = k(ξ t , x t , v) affine in ξ t , with v ∈ R, defined on U and a stochastic coordinate transformation z t = Φ(x t ) defined on U such that the closed-loop systeṁ in the coordinates z t = Φ(x t ), is linear, deterministic and controllable.
Remark 9. By requiring that the control u t = k(ξ t , x t , v) is an explicit function of the white noise, as in the statement of Problem 1, we may be enlarging the class of systems (11) beyond what could be allowed. In the general case where k is any nonlinear function of ξ t , the resulting closed-loop system in the derivative notation, i.e. (10), is not anymore equivalent to the one in the differential notation, i.e. (11). In fact, this equivalence is preserved only when the resulting closed-loop dynamics is affine in the white noise. Therefore, we hereby define an idealistic control law to be admissible if the dynamics of the closed-loop system is affine in ξ t . This is achieved easily by requiring that k is affine in ξ t as done in the statement of Problem 1. Note that the idealistic controls that we introduce in this paper are all admissible by construction.
Proposition 3. Problem 1 is solvable if and only if there exists a real valued function h(x t ) such that system (16) with the output y t = h(x t ) has stochastic relative degree n atx.
Proof. Sufficiency. Suppose y t = h(x t ) is such that the stochastic relative degree of system (16) is n atx. Then by Proposition 2, there exists a change of coordinates for all x in a neighbourhood U ofx, such that the normal form of system (16) iṡ By the definition of stochastic relative degree, there exists a neighbourhood U ofx such that b(z t ) = 0 in Φ(U ). Therefore the control law is well-defined in Φ(U ). Therefore, in the neighbourhood U of where A and B are such that B AB . . . A n−1 B has rank n.  (12) and then use Lemma 1 to show that these arguments hold for the Itô system (11) as well.
Remark 10. As the coefficient c is in general a function of ξ t , the feedback linearising control (17) could require the knowledge of the exact value of the white noise for all t, because this is the only way that the noisy dynamics of the open loop system (16) can be rendered deterministic via feedback. Of course, the assumption that the noise can be used in the feedback loop is unrealistic. However, building on the results of this section, in Section V we design a practical hybrid control scheme. This hybrid scheme implements a deterministic state feedback law which periodically compensates, in an approximate way, the noise.
Remark 11. The control u t = k(ξ t , x t , v) designed in the proof of Proposition 3 is affine in the variable ξ t , because in (17) the coefficient b is not an explicit function of ξ t , whilst the coefficient c is affine in ξ t . Therefore, the feedback linearising control is always admissible because replacing its expression in (11) leaves the closed-loop dynamics affine in ξ t .
To conclude this section, we provide necessary and sufficient conditions for the existence of an output y t = h(x t ) which makes the stochastic relative degree of system (16) equal to n at a pointx. Theorem 1. Problem 1 is solvable if and only if 1) the matrix g(x) ad f S g(x) . . . ad n−1 f S g(x) is invertible and 2) the distribution span{g, ad f S g, . . . , ad n−2 f S g} is involutive nearx.

Proof. By Proposition 3, Problem 1 is solvable if and only if
it is possible to find an output function h(x t ) satisfying the conditions in Definition 6. By Lemma 1, this amounts to the set of conditions 1) L g L k f S h(x) = 0 for all x in a neighborhood ofx and all k ∈ {0, ..., r − 2}.
2) L g L r−1 f S h(x) = 0, which is a set of partial differential equations and a nontriviality condition. By If the stochastic relative degree r of system (11) is strictly less than n atx, we indicate by ζ t = z 1 . . . z r the first r components of the transformed state z t = Φ(x t ) and by η t = z r+1 . . . z n the remaining n − r. Note that if r < n the control law (17) linearises only the dynamics of ζ t , thus we say that it partially feedback linearises system (15).
The zero dynamics of a nonlinear stochastic system is, analogously to the deterministic case, the dynamics of the internal variable η t when the input and the initial conditions are chosen in such a way that the output is constrained to be identically zero. This is achieved by setting ζ 0 = 0 and u z,t = −c(ξ t , 0, η t )/b(0, η t ). Note that the control u z,t is affine in ξ t , hence admissible. We now provide a definition that extends the concept of zero dynamics to nonlinear stochastic systems of the form (11). Definition 7. (Zero Dynamics) The stochastic differential equationη is called the zero dynamics of system (11).
The properties of the zero dynamics are fundamental in studying the problem of asymptotic output tracking, which is the topic of Section VI.

V. PRACTICAL LINEARISATION VIA STATE FEEDBACK
In this section we introduce a causal method to estimate the sequence of variations of the Brownian motion between successive time instants. We then show that the use of these estimates is beneficial in the design of practical feedback linearising controls.

A. Estimation of the Brownian motion
The estimation of the Brownian variations is performed by periodically sampling the state. This method was first introduced in [33] in order to practically solve the problem of output regulation of linear stochastic systems. We now extend it to the present context of nonlinear stochastic systems.
Let {t k } k∈Z ≥0 be a sequence of equally-spaced sampling times, with t k − t k−1 = ε for all k ∈ Z >0 . Define the differences ∆W ε (k) = W t k − W t k−1 and ∆x(k) = x t k − x t k−1 . Our aim is to show that it is possible to compute a causal estimate ∆ W ε (k) of the quantity ∆W ε (k) by comparing the samples of the state of the system at times t k−1 and t k . In particular, we want this estimate to "converge", in a sense to be defined, to the stochastic differential d W t as the sampling period ε converges to zero. Let L I be the space of functions that are integrable in Itô's sense. Then with the notation ∆ W ε ε − → d W t we mean that for all α ∈ L I lim ε→0 k α(t k−1 , w)∆ W ε (k) = t 0 α(τ, w)dW τ for almost all w ∈ ∇. For ease of notation, define F t k = f (x t k ) + g(x t k )u t k and L t k = l(x t k ), which are the drift and diffusion coefficients, respectively, of system (10) evaluated at time t k . We make the following assumption.
The rationale of this assumption is explained later, in Remark 12 at the end of Section V. Under Assumption 2 the Moore-Penrose left pseudo-inverse of L t k , i.e. L † t k = (L t k L t k ) −1 L t k , is well-defined almost surely. The following Lemma extends the results in [33] to systems with nonlinear drift and diffusion terms. Lemma 3. Consider system (11) and let Assumption 2 hold. Let {∆ W ε (k)} k>0 be a sequence of scalars defined as Proof. Let k ∈ Z >0 . By [34, Theorem 7.1] holds, where o(ε 2 ), which is the one-step truncation error of the forward-Euler scheme, is an infinitesimal of the same order of ε 2 . The previous expression can be rewritten as Since L t k−1 has full column rank almost surely, the expression holds almost surely. Defining ∆ W ε (k) as in (18) yields . almost surely. Let α t ∈ L I . Then Taking the limit of both sides as ε tends to zero yields ∆ W ε ε − → dW t , since for all α ∈ L I lim ε→0 k

B. Hybrid control law
In this section we discuss how the sequence of estimates {∆ W ε (k)} k can be used to design a practically implementable hybrid feedback control law that approximates the idealistic input (17), namely u lin t = (−c(ξ t , z t ) + v)/b(z t ), which linearises the dynamics of the first r components of system (15). We show that the accuracy can be improved by increasing the sampling time ε. We formally state the problem as follows.
Problem 2. (Practical Feedback Linearisation) Consider system (15) and let z lin t be its solution when u t = u lin t . The problem of practical feedback linearisation consists in finding a control lawû lin t (ε), depending on the sampling rate ε, such that, if z t is the solution of (15) when u t =û lin t , then for every σ > 0 The meaning of Problem 2 is to find a causal controller for which the closed-loop system dynamics converges in probability to the idealistic closed-loop dynamics as the sampling time is made smaller. To begin with, since the coefficient c is affine in ξ t , we can express it as c(ξ t , z t ) = c d (z t ) + c s (z t )ξ t for some mappings c d and c s . Note that since in practice ξ t is not known, a naive approximation of ξ t boils down to replacing it with its expectation, i.e. zero. Therefore, a first causal approximation of the coefficient c is given by only the term c d (z t ), in turn implying that the control u lin t can be practically approximated by the naive law We call this basic feedback law the zero-noise control. By replacing c s (z t )ξ t with zero in the expression of c(ξ t , z t ), the zero-noise control does not perform any form of stochastic compensation when performing feedback linearisation. Consequently, there is no guarantee that the closed-loop behaviour of system (15) with u t = u zn,lin t is any close to its idealistic behaviour, i.e. when u t = u lin t . The goal of this section is to improve the performance of the zero-noise control u zn,lin t by leveraging the estimated sequence {∆ W ε (k)} k and show that we can recover the idealistic behaviour in probability. Therefore, we defineû lin t = u zn,lin t +u s t , with u s t to be specified in such a way thatû lin t is a "better" approximation of u lin t than u zn,lin t . By replacing the control u lin t in (15), the dynamics of the transformed system becomeṡ where the term c s (z t )ξ t in the dynamics of the r-th component is due to the fact that the approximating control u zn,lin t cannot cancel the noisy dynamics as the idealistic control u lin t does. We now want to design the control u s t employing the estimates {∆ W ε (k)} k introduced in Section V-A, to reduce the contribution of the term c s (z t )ξ t onto the dynamics of the system. Since the quantity ∆ W ε (k) carries information on the evolution of the noise between t k−1 and t k , we look at the evolution of z r between these two consecutive sampling times. The value of z r at time t k is given by is the contribution of the noise on the dynamics of z r between the two sampling times. Our goal is to minimise this contribution using u s t and the estimate ∆ W ε (k) obtained at time t k . The fact that ∆ W ε (k) is only available a posteriori, at the end of the sampling period, suggests that u s t should induce a jump variation at time t k in the state z r in order to compensate for the quantity β d (k). In other words, the dynamics of the closed-loop system should be hybrid. It is then necessary to introduce a simplified jump notation. At time t k , we denote by z t k the state before the jump, and by z t + k the state after the jump. The flow dynamics of the closed-loop hybrid system we seek is, therefore, given bẏ for all t ∈ R ≥0 , while the jump dynamics is given by for all k ∈ Z >0 , where {u * (k)} k is a yet to be defined sequence of scalars depending on {∆ W ε (k)} k . Alternatively, the aforementioned hybrid dynamics (21)- (22) can be equivalently produced by using in (20) an impulsive control u s t given by where δ(t) is a Dirac delta. As a result of the hybrid dynamics induced by the control u s t , the expression of z r before the jump at time t k is given by while after the jump by Thus, we have reduced the problem of approximate partial feedback linearisation to the problem of finding the sequence {u * (k)} k such that the contribution of the term β d (k) + b(z t k )u * (k) is minimised. In particular, we look for a sequence {u * (k)} k which retrieves the exact linearisation of the dynamics of ζ t as ε tends to zero. We are now ready to solve Problem 2.
Proof. It is trivial to observe that, since system (15) is in normal form, (19) holds if and only if it holds for the r-th component of the state z t . Thus, we now focus on the r-th component. Under the controlû lin t , the jump of z r at t k is given by thus, in light of the previous discussion, This can be rewritten as Now, consider any t ∈ (t k−1 , t k ), for some k, and let 0 <ε < ε be given byε = t − t k−1 . Then which can be discretised with the Euler-Maruyama method as with ∆Wε(k) = W t − W t k−1 , and where o(ε 2 ) is the one-step truncation error, which is of orderε 2 . Similarly, the discretised dynamics of z lin r,t is z lin Observe that, by the properties of the Brownian motion, ∆Wε(k) is a normally distributed random variable with zero expectation and varianceε. Moreover c s is bounded near zero, because it is the Lie derivative of a smooth function. By taking the limit of (24) as ε (henceε) goes to zero and using (23), we have that for every σ hence the claim follows.
The previous proposition states that, when u s t is selected as the controlû lin t approximates the idealistic control u lin t as ε tends to zero, in the sense that the dynamics of the variable ζ t can be made approximately linear with an accuracy increasing as ε decreases.
Remark 12. The rationale of Assumption 2 is that the noise is persistently exciting, so it can be estimated by measuring the state of the system. From a technical viewpoint, this assumption is necessary for the development of the theory because it ensures the boundedness of L † t k . However, Assumption 2 should not be considered practically restrictive. In fact, if |L t k | <δ, for an arbitrarily smallδ, this would imply that the noise contribution to the dynamics of the system is sufficiently small at time t k to be considered negligible. In that case it is simply possible to avoid performing the stochastic compensation at time t k and still obtain good control performance.

C. Remarks on the control input in the diffusion: m = 0
At the beginning of Section IV we made the standing assumption that m(x) ≡ 0 nearx, thus restricting the class of systems considered to those with the control input appearing only in the drift term of the stochastic differential equations. This assumption allowed us to obtain sharper analytical results. In this section we discuss the rationale of such assumption from a control viewpoint, which has implications both in the idealistic and the practical frameworks.
As shown in (9), for general m, the dynamics of the r-th component of the transformed state z t = Φ(x t ) is a quadratic function of the control input. This implies that, in the case that a(z t ) = (1/2) m G m l S r−1 f h(Φ −1 (z t )) is not zero for x near x, the idealistic feedback linearising control would have the form

which has real values if and only if the input
The corresponding zeronoise control would have the form It is evident that, although in both cases the control input can be forced to be real by an appropriate choice of v, from a practical viewpoint such a choice may not leave any space for a control law achieving objectives such as, for instance, output tracking. Moreover, even though we may be able to define a zero-noise control, the construction of a hybrid controller presents even more challenges, as the quadratic u lin t is not affine in ξ t . Of course, one might make the standing assumption that a(z t ) ≡ 0 for x nearx. This class of systems would include the systems for which m ≡ 0, which we address in detail in this paper, but also those systems with m = 0 with the additional assumption that m G m l S r−1 f h(Φ −1 (z t )) = 0. However, the design of controllers poses substantial technical challenges even in this case. For the sake of completeness, we briefly give an overview of these issues. Suppose m = 0 and a(z t ) ≡ 0 for x nearx. The coefficient b in (9) is therefore affine in ξ t , i.e., b(ξ t , z t ) = b d (z t ) + b s (z t )ξ t . In the idealistic framework, a feedback linearising control would require a division by b(ξ t , z t ). Unless further, possibly restrictive, assumptions are made, such a division may result in a non-admissible control. Additionally, the fact that the coefficient b depends on the noise affects impulsive compensations as well. In fact, the dynamics of z r when the controlû lin t is applied iṡ Thus any compensating control u s t inevitably introduces noise at time t + k which cannot be compensated for because ξ t k is a random variable which is independent of the process ξ t , t < t k . This makes it impossible to conclude on the convergence in probability of z r,t + k to z lin r,t k as ε approaches zero, because the random variable ξ t k can take arbitrarily large values in R with nonzero probability.

VI. ASYMPTOTIC OUTPUT TRACKING
In this section we first design an idealistic controller, i.e. using the white noise in the feedback loop, to make the output asymptotically track a reference signal. Then we show that a practical feedback control law leveraging the causal stochastic compensations introduced in Section V is able to retrieve the idealistic result, in the limit of the sampling time ε going to zero.
We start by introducing the idealistic control and we show that, under suitable stability hypotheses on the zero dynamics, it is possible to control the system so that its output tracks reference trajectories while its internal variables remain bounded almost surely. First observe that, as long as z t = Φ(x t ) is chosen such that q ≡ 0 near zero (which is always possible, see Section III-A), the zero dynamics of system (15) is affine in ξ t . Let it be expressed aṡ Consider a reference signal y R which is continuously differentiable r times with values in a neighbourhood of zero. We assume that the initial state of the transformed system (15) is arbitrary while in a neighbourhood of zero and we seek a feedback control u t that makes the output y t of the system asymptotically converge to y R . Let v(ζ t , y R (t)) = y with d i ∈ R for i = 0, . . . , r − 1 to be determined, and the admissible idealistic feedback control law be given by Define the tracking error e t := y t − y R (t). Then the control u track t forces the dynamics of the tracking error to be e (r) t + d 0 e t , which can be made exponentially stable by selecting the coefficients d i such that which is the characteristic polynomial of the matrix has roots with negative real parts. We also study the boundedness of ζ t and of the internal variable η t under the control u track t , when y R and its first r − 1 derivatives are bounded. Define ζ R (t) = y R (t) . . . y Theorem 3. Consider system (15). Suppose y R (t), y and let p d and p s be Lipschitz continuous. Moreover, assume that there exists a strict Lyapunov function V (η R , t) for (28) such that ∂V ∂η R,i (x, t) and are bounded for all x in a neighbourhood of the origin and t ≥ 0. Suppose that the roots of the polynomial Λ(s) in (26) have negative real part. Then for sufficiently small R > 0, if then for all > 0 there exists δ > 0 such that almost surely, i.e. the response z i and η t , t ≥t ≥ 0, of system (15) under the control law u track t is bounded almost surely.
Proof. System (15) under the control law u track t can be rewritten in the forṁ with A given by (27), which has characteristic polynomial Λ(s). Therefore θ t has asymptotically stable dynamics. Let ν t = η t − η R,t and P (ξ t , ν t , θ t , t) = p(ξ t , ζ R (t) + θ t , η R,t + ν t ) − p(ξ t , ζ R (t), η R,t ). Note that the systeṁ is in the form (36) (see Appendix B). Moreover, system (30) additionally satisfies the hypotheses of Lemma 8 in Appendix B because of the assumption of Lipschitz continuity of p d and p s and of the existence of V as in the statement. Then, by Lemma 8, (0, η R,t ) is an almost surely uniformly stable solution of (29) and the claimed estimates follow.
The previous theorem solves the idealistic local asymptotic output tracking problem, i.e. the output y t = z 1 asymptotically converges to y R whilst the state z t remains bounded almost surely.
We now focus on practical output tracking. While it can be proved (see Proposition 4 in Appendix C) that under some technical assumptions a zero-noise control is sufficient to asymptotically stabilise the equilibrium at the origin, a controller not performing any sort of compensation for the stochastic disturbances does not guarantee asymptotic tracking. This is because if the system tracks a non-zero reference, then the states will not converge to zero and so the noise will enter the dynamics in a persistent fashion. Hence the noise might drive the states away from the desired trajectory, possibly inducing instability. On the contrary, the control laŵ and u s t given by (25) is able to prevent this when ε tends to zero by approximately compensating for the Brownian-induced disturbances, as shown in the following result. Corollary 1. Consider system (15) and suppose that Assumption 2 and the assumptions in Theorem 3 hold. Then the control lawû track t is such that y t converges to y R in probability and the state z t , t ≥t ≥ 0, of system (15) is bounded in probability in the limit as ε approaches zero.
Proof. By Theorem 3 the control u track t makes the output y t of (15) asymptotically converge to y R while keeping the internal states bounded. Let z track t and z t be the the state of system (15) when u t = u track t and u t =û track t are applied, respectively. Then, by Theorem 2, for every σ > 0, lim ε→0 P( z t − z track t ≥ σ) = 0, and the claim follows.

VII. ILLUSTRATIVE EXAMPLE
In this section we illustrate the validity of the theory by means of a numerical example. Consider the following nonlinear stochastic system in the form (11) with and s i and c i denoting sin x i and cos x i respectively. Let the output of the system be y t = x 1 + s 2 − x 3 . We are interested in analysing the system around the origin, i.e. we setx = 0. The goal is to bring the system to its normal form and to perform asymptotic output tracking. The first step is determining the stochastic relative degree of the system at zero. We set z 1 = y t = x 1 + s 2 − x 3 and we compute its derivative applying Itô's formula (we omit the procedure for brevity), thus obtaining dz 1 = (−2x 3 −s 2 )dt. As neither the input u nor the noise appear in this expression, the stochastic relative degree, if defined, is higher than one at the origin. We set z 2 = −2x 3 − s 2 and, by computing its derivative, we obtain The system has therefore stochastic relative degree r = 2 at the origin. Moreover, Setting z 3 = x 1 − x 3 makes the coordinate change z = Φ(x) a diffeomorphism in a neighbourhood of the origin, with In this new set of coordinates the dynamics of the system is given bẏ which is in the stochastic normal form. The zero dynamics of the system is obtained by equating z 1 = 0 and z 2 = 0, z 3 = η t , which yields x 1 = (3/2)η t , s 2 = −η t . Replacing these in the third equation in (31) we get the zero dynamics as followṡ Its first approximation around the origin isη t = −2η t +3η t ξ t = Aη t + F η t ξ t , which is asymptotically stable almost surely because A − F 2 /2 < 0. Therefore the zero dynamics of the system is almost surely asymptotically stable. We now choose a reference signal of the form y R (t) = β + α cos(ωt) and illustrate that u track t achieves asymptotic output tracking. To this end, we first performed a simulation in the idealistic scenario in which the noise is used in the feedback law. We selected the coefficients d 0 = 12 and d 1 = 7 in the input v so the characteristic polynomial Λ(s) in (26) has roots in −3 and −4, thus guaranteeing asymptotic stability of the linearised sub-system in the coordinates ζ t . The reference signal y R is characterised by β = 0.1, α = 0.01 and ω = 5. The nonlinear stochastic differential equations were integrated using the Euler-Maruyama scheme with period ∆t = 10 −6 . In Figure 1 we show the time history of the state in the coordinates z t when u track t is applied. Observe that the first two components (blue and red lines) display, as expected, linear and deterministic behaviours, while the internal variable z 3 has noisy dynamics. Because of the properties of the zero dynamics, z 3 stays bounded under u track t . Moreover, the component z 1 (blue line), which by the definition of normal form is the output y t of the system, asymptotically converges to the reference signal y R (purple/dashed line).
We now turn our attention to practically realisable controllers. To this end, we first discuss the numerical implementation of the simulations in the cases where stochastic compensations are performed with a period ε. The continuous-time dynamics was integrated, as usual, using the Euler-Maruyama numerical scheme with fixed period ∆t = 10 −6 . The stochastic compensations were performed with a period ε, which must necessarily satisfy ε > ∆t. We performed simulations with values of ε of 10 −3 , 10 −4 and 10 −5 , in order to illustrate the limit behaviour of the solutions as ε decreases. Note that it could be possible to select smaller ε as long as ∆t is decreased accordingly.
We illustrate in Figure 2 that when the control law with compensations is employed and ε is decreased, the trajectory of the state under the idealistic control is retrieved. In Figure 2 we consider again the output tracking setting and display the time history of the component z r = z 2 under the controls u zn,track t (blue line),û track t , with varying ε (10 −3 (red line), 10 −4 (yellow line), 10 −5 (purple line)), and u track t (green line). Observe that, since the controlû track t improves the noise compensations as ε decreases, the trajectories of z 2 under the compensated control tend to the idealistic trajectory.
To conclude, Figure 3 shows a comparison between the time histories of the tracking error z 1 − y R = y t − y R when the controls applied are, respectively, u zn,track t (blue line) orû track t with decreasing values of ε, namely 10 −3 (red line), 10 −4 (yellow line), 10 −5 (purple line). While the hybrid controller produces, for any of the values of ε, strikingly better asymptotic performances than the zero-noise control, observe that the tracking error is made smaller and smaller as ε is decreased. This is in line with the fact that the component z 1 = y t under u track t tends, as ε approaches zero, to z 1 under the control u track t , which in turn asymptotically approaches y R .

VIII. CONCLUSIONS
In this paper we have addressed the path-wise control of nonlinear stochastic systems. First we have introduced a notion of stochastic relative degree and normal form. Then, leveraging these, we have presented a feedback linearising controller. We have observed that this controller is not causal, hence referred to as idealistic, because it requires a feedback of the noise. To overcome this limitation, we have introduced a hybrid control architecture that estimates the Brownian motion from measurements of the state and uses these estimates to periodically compensate, in approximate way, for the noise. We have proved that the performances of the idealistic control law are retrieved when compensations at a frequency tending to infinity are performed. Finally, we have solved the problem of asymptotic output tracking, both in the idealistic and practical framework, and we have provided an illustrative example.
B. Almost sure stability of perturbed stochastic systems Consider the stochastic time-varying systeṁ x t = f d (x t , t) + f s (x t , t)ξ t =f (ξ t , x t , t), (33) for which we introduce the following Lipschitz assumption [35].
Assumption 3. f d and f s are locally Lipschitz continuous for all t ≥ 0.
We now introduce a concept of almost sure total stability for nonlinear time-varying stochastic system. Consider the following extension of the definition of strict Lyapunov function for stationary stochastic systems given in [36] to the case of time-varying stochastic systems.
Definition 8. Consider the autonomous system (33). A smooth function V : U × R ≥0 → R, where U is a bounded domain, is said to be a strict Lyapunov function for system (33) if V (0, ·) = 0 and W 1 and W 2 are continuous positive definite functions on U ; • there exists a positive definite function W 3 (x) such that ∂V ∂t (x, t) + L f d V (x, t) + 1 2 fs G fs V (x, t) < −W 3 (x) and L fs V (x, t) = 0 for all x ∈ U , t ≥ 0.
It is possible to show that the existence of a strict Lyapunov function for system (2) is sufficient to conclude the almost sure (uniform) asymptotic stability of its equilibrium point (as defined, e.g., in [37], [38]). Namely, the following result is an extension of [36, Theorem 2.5].
Lemma 6. Under Assumption 3, if there exists a strict Lyapunov function for system (33), then the equilibrium point at the origin of system (33) is almost surely uniformly asymptotically stable.
The proof follows by replacing in the proof of [36, Theorem 2.6] the time-invariant infinitesimal generator of V with its time-varying version.
Remark 13. By dropping the condition that L fs V (x, t) = 0 in Definition 8, the results can be reformulated in the context of the weaker stability in probability. See [27,Theorem 11.2.8] for an equivalent of Lemma 6 in this context.
Consider now a persistently perturbed version of system (33), namely dx t = (f d (x t , t) + p d (x t , t))dt + (f s (x t , t) + p s (x t , t))dW t (34) where p d and p s are Lipschitz. In the following we formalise the idea that if p d and p s are small enough perturbations, then the stability properties of (33) are analogous to those of (34). In other words, we provide an extension to stochastic systems of the concept of total stability presented, e.g., in [39,Definition 56.1]. To this end, letp(ξ t , x t , t) = p d (x t , t) + p s (x t , t)ξ t . Definition 9. Consider the stochastic systems (33) and the perturbed systeṁ Suppose that there exists a solution x p,t (t,x) of (35). The equilibrium x = 0 of (33) is said to be totally stable almost surely if for each > 0 there exist positive δ 1 ( ), δ 2 ( ) and δ 3 ( ) such that x p,t (t,x) < for all t ≥t almost surely provided that x < δ 1 , p d (x, t) < δ 2 and p s (x, t) < δ 3 for all (x, t) satisfying t ≥t and x ≤ .
The following lemma provides some properties that a strict Lyapunov function must satisfy in order for system (33) to have an almost surely totally stable equilibrium at zero. Lemma 7. Consider systems (33) and (35). Suppose that Assumption 3 holds. If there exists a strict Lyapunov function V for system (33) such that • ∂V ∂xi (x, t) and ∂ 2 V ∂xi∂xj (x, t) are bounded for all x ∈ U and t ≥ 0, • L ps V (x, t) = 0 for all x ∈ U and t ≥ 0, then system (33) has an almost surely totally stable equilibrium at x = 0.
Proof. The proof follows from [39,Theorem 56.3] by noticing that the time derivatives of V for (33) and (35)  Finally, the following lemma gives important stability properties of nonlinear time-varying stochastic systems driven by almost surely stable systems. Lemma 8. Consider the stochastic systeṁ y t =ã(ξ t , y t , z t , t) =ã d (y t , z t , t) +ã s (y t , z t , t)ξ t , z t =b(ξ t , z t ) =b d (z t ) +b s (z t )ξ t , and assume the following. A1) (y, z) = (0, 0) is an equilibrium of (36) andã d andã s are Lipschitz continuous for all t ≥ 0 in a neighbourhood of zero. A2) There exists a strict Lyapunov function V (x, t) for the first equation of system (36) such that ∂V ∂xi (x, t) and ∂xi∂xj (x, t) are bounded for all x ∈ U and t ≥ 0. A3) The equilibrium z = 0 ofż t =b(ξ t , z t ) is stable almost surely.
C. Almost sure stability under zero-noise control Lemma 9. Consider the systeṁ