State Estimation for Linear Dynamic System With Multiple-Step Random Delays Using High-Order Markov Chain

To cope with the large state estimation error due to sensor delay, a novel flexible model is explored to describe a linear dynamic system with multiple-step random delays in this paper. Compared with existing models, this model is more consistent with the actual situation. Based on the new model, the main difficulty, which is to determine the probability of any number of steps delayed, is overcome by applying techniques of high-order Markov chain. Then, the Kalman filtering problem with measurement delays is converted to random parameter matrices Kalman filtering(RKF), the new approximate state estimators are proposed. For a <inline-formula> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula>-step random delay model, we prove that it can be treated as a <inline-formula> <tex-math notation="LaTeX">$(2n-1)th$ </tex-math></inline-formula>-order Markov chain, making it theoretically feasible to apply the method in this paper to deal with any multiple-step delay model. Some illustrative numerical examples are presented to demonstrate the efficiency of the new model and superiority over existing algorithms.


I. INTRODUCTION
In practice, data received from the sensors may be randomly delayed due to many uncertainties, such as sensor temporal failures, sensor saturated mechanisms, equipment failure and heavy network traffic, causing out-of-order measurement sequences [1]- [3], which greatly reduces the performance of traditional Kalman filter in state estimation. Both in linear and nonlinear systems, there have been a lot of research interest in the state estimation problem with random delays owing to its significant applications in fields such as signal process, GPS integration navigation, target tracking, and radar control [4]- [8].
The theory of random parameter matrix models is usually used to deal with state estimation. To mention a few, Yaz [9] considered a general discrete-time stochastic bilinear system model and derived the mean square optimal linear unbiased estimator for random parameter matrix models. Nahi [10] gave the minimum mean-square estimators of recursive form for two different types of uncertain observation problem, in which methods and techniques in this paper The associate editor coordinating the review of this manuscript and approving it for publication was Liang Hu . have been used for references by many following researches. Chen et al. [11] derived several conclusions on Kalman filtering for conditionally Gaussian systems with random matrices. In addition, quite a few researchers proposed studies on systems with random parameter matrices in NaNacara and Yaz [12], Shen et al. [13], etc. In recent years, based on random parameter matrix models, many models and techniques have been proposed to handle random delay problems. Many literatures only take one-step delay into consideration, but the actual situation is that multiple-step random delays often occur. Yaz and Ray [14] proposed a classical and concise model to characterize the nature of sensor delay and gave unbiased minimum variance state estimators. A least square linear estimator was investigated based on the covariance information approach [15]. Furthermore, Linares et al. [16] and Caballero-Águila et al. [17] extended the result to multiple-step random delays and multiple-step packet dropouts, respectively. It is worth mentioning that the random delay was traditionally defined by a Bernoullidistributed sequence with a deterministic parameter, see e.g. [2], [14], and [18]- [21]. Luo et al. [22] derived the recursive state estimation of the random parameter matrices Kalman filtering (RKF). In this paper, the new state estimators are proposed by RKF after we use a delay model to tranform the original problem into the form that RKF can deal with. However, there is a significant problem that a delay model widely used in many previous studies, as in [14], does not quite fit the actual situation, which will be explained in detail in the next section. Then, a novel and more realistic model is explored to describe a linear dynamic system with multiplestep random delays.
In the past few years, the high-order Markov chain has also attracted much attention. In terms of mathematical theory, there are many literatures on high-order Markov chain. Raftery conducted a series of researches on a special kind of high-order Markov chain [23]- [25]. Adke and Deshmukh [26] obtained the limit distribution of the highorder Markov chain under conditions weaker than those assumed by Raftery. Several other special high-order Markov chains are also frequently studied in Pegram, Jacobs and Lewis [27] and Logan [28]. The high-order Markov chains discussed in the above studies are all special and conform to specific rules. We use the definition of generalized higherorder Markov chain in this paper, not focusing on its profound mathematical properties, the main purpose is to apply it to solving the key problem in state estimation. From the perspective of practical application, the application of high-order Markov chain in various fields such as signal processing and data transmission has been widely studied. Myint et al. [29] investigated a second-order Markov-based 5G error model to model the underlying error process in the 5G network. Wang et al. [30] proposed a variable-order Markov model based on kernel smoothing, to apply to the analysis of GPS trajectories. Liu et al. [31] applied Markov chain to model the case under the assumption that the random communication delays exist both in the system state and in the mode signal. In this paper, we also introduce high-order Markov chain, and define the expression of its state space, then use it to express our designed model in mathematical language. However, the high-order Markov chain is not suitable for derivation and calculation, thus we transform it into the standard Markov chain by redefining(expanding dimension) the state space, making it possible to calculate the probabilities of delay, which is the key part of extending one-step delay model to multiple-step delay model.
Pang et al. [32] explored a novel and more elaborate model to describe a linear dynamic system with random delays and presented state estimators. However, this model only works under the assumption of one-step delay. In order to extend the previous research results to multiple-step random delays, a model is proposed in this paper to reasonably describe a linear dynamic system with multiple-step random delays, in which the model in Pang et al. [32] is a special case of our model under the assumption of one-step delay. The high-order Markov chain is used to process the multiplestep delay model. It is proved that any n-th order Markov chain can be transformed into a first-order Markov chain by redefining the state space. Then, based on high-order Markov chain, the Kalman filtering problem with measurement delays is converted to random parameter matrices Kalman filtering (RKF). We prove that a n-step random delay model corresponds to a (2n − 1)th-order Markov chain. Treating the converted problem as RKF, the new approximate state estimators are proposed. The numerical examples show that these new estimators work better than the existing algorithm in many cases.
This paper is organized as follows. In Section II, the new model featuring n-step random delay characteristics is presented, and high-order Markov chain is introduced. In Section 3, in the case of 2-step delay, the model is transformed into the form of random parameter matrices Kalman filtering(RKF) by applying third-order Markov chain. Moreover, two approximate estimators for the linear dynamic system are given. In section 4, in the case of n-step delay, an important theorem on the order of the Markov chain corresponding to the n-step delay model is proved, which enables us to process any multiple-step delay models through the same approach as in section 3. In section 5, numerical examples are provided. Finally, a summary of our work is presented and possible future studies are discussed in Section 6.

II. PRELIMINARY
In this section, we will introduce some preliminaries.

A. MULTIPLE-STEP DELAY MODEL
Consider a linear dynamic system with state-space description: where F k and H k are n × n, q × n constant matrices, respectively, with q ≤ n, {ν k } and {ω k } are system and observation noise sequences, respectively, with known statistical information such as zero-mean and covariance matrices as follow: To obtain optimal state estimators of the system with random sensor delay, it is prerequisite to build a model that delicately describe the nature of random delay. When random delay exists, the measurement received by the filter at time k, denoted as y k , may not equal to sensor output z k . The universal measurement equation widely used in many former studies is as follow, assuming that an observation may delay up to n steps: k } is a sequence of independent Bernoulli random variables(not necessarily i.i.d.). However, this equation does not match the actual situation. For instance, under the assumption that a measurement may delay one step at most, the equation comes to y k = ζ 1 k z k + (1 − ζ 1 k )z k−1 . If ζ 1 k = 0, then y k = z k−1 , indicating that the sensor output is delayed by one step. Then at next step, y k+1 must equal to z k , or z k would VOLUME 8, 2020 be lost and this measurement equation would be invalid. The model works worse when more steps delayed. Apparently, this is not true in practice.
In this paper, a new model is proposed. To begin with, a few definitions are given, as follows.
Definition 1 (Delay): Under the assumption that the time stamp of the sensor output z k is unknown or inaccurate for the filter, the sensor output z k is called delayed when z k arrives at the filter later than some sensor output z n , (n > k).
Definition 2 (n-Step Delay): If the sensor output z k arrives at the filter after sensor output z k+n , the k-th sensor output can be called n-step delayed.
Consider a discrete time dynamic system (1) -(2) with n-step random delays. The following is assumed.
Thus, the model in Pang et al. [32] is a special case of our model where n = 1.

B. MAIN CHALLENGE
Denote τ as the order of sensor outputs that arrive at the sensor. Thus, at order τ , 2n + 1 observations may arrive at the sensor. Denote the probability that y τ = z τ −n , y τ = z τ −n+1 , · · · , y τ = z τ +n as p −n τ , p −n+1 τ , · · · , p n τ respectively. The measurement equation at order τ can be described as: If the probability p i τ (i = −n, · · · , n) is known, we can estimate the state of (1) and (2) by RKF or RKF with finitestep correlated noises. The key point here is to determine how to calculate the probability p i τ (i = −n, · · · , n) in (4). Remark 1: For the one-step delay problem (n = 1), the probability p i τ (i = −1, 0, 1) can be calculated recursively. Especially, when the delay probability β of each step is constant, this model can be treated as a first order Markov chain, where π τ +1 = (p −1 τ +1 , p 0 τ +1 , p 1 τ +1 ) = π τ P (see [32]). However, this method is invalid in a multiple-step model.

C. HIGH-ORDER MARKOV CHAIN
For the one-step delay problem (n = 1), the probability p i τ (i = −1, 0, 1) can be calculated recursively. Especially, when the delay probability β of each step is constant, this model can be treated as a first order Markov chain, where π τ +1 = (p −1 τ +1 , p 0 τ +1 , p 1 τ +1 ) = π τ P (see [32]). However, for a multiple-step delay model, P depends not only on the current step, but also on previous steps. Therefore, high-order Markov chain is introduced as a tool to process the multiplestep delay model.
The following lemma shows that any n-th order Markov chain can be transformed into a first-order Markov chain by redefining the state space(dimension extension).
Lemma 1: For any m-th order Markov chain {X n } with state space X , {X n } can be transformed into a first-order Markov chain {X n }(n ≥ m) with state space X m , which is a m-th power Cartesian product of X .
Proof: For n ≥ m, define X n as a m dimensional vector (X n−m+1 , X n−m+2 , · · · , X n ). Thus, X n ∈ X m . By Definition 3, From (6), {X n }(n ≥ m) is a first-order Markov chain, with state space X m . We call {X n }(n ≥ m) the corresponding firstorder Markov chain of {X n }.
For the sake of notation simplicity, detailed procedures of iteratively calculating p i τ (i = −2, · · · , 2) using third-order Markov chain in 2-step delay model will be proposed in section 3. In section 4, an important theorem is given, which enables us to generalize it to n-step delay without difficulty.

III. 2-STEP DELAY
Based on Assumption 1, the hypothesis tree of the two-step delay model is presented in Fig. 1. It can be observed that the sensor measurement at y τ relates to several previous states, which reminds us of high-order Markov chain.
Apply the high-order Markov chain to describe our new model. Define m τ as the corresponding random variable of the observation at order τ as follow: It is obvious that {m i } is a high-order Markov chain. Moreover, {m i } is a third-order Markov chain from the theorem in section 4, indicating that the value of any {m i } relates     (8) which has only 12 elements. Thus, the 12×12 transition matrix of {m i } from m k to m k+1 , denoted as P k , can be obtained, as follows: where P k (i, j) denote the i-th row and j-th column element of the transition matrix P k . Except for the above elements, all the other elements in P k are 0.
It can be seen that the transition matrix P k matrix is sparse, which reduces the computational burden. Also, if β 0 i , β 1 i (i = 1, 2, · · · ) are constant, then {m i } is positive recurrent, aperiodic, irreducible and has invariant distribution π .
(13) Thus, (12) immediately. By Lemma 2, the measurement equation at time point τ is which can be written as where (1) and (2), Thus, y τ = ξ τ Z τ is equivalent to where Let E(ξ τ ) = ξ τ , so ξ τ = ξ τ + ξ τ . Substitute it into (16), then the measurement equation can be transformed into where Next, two approximate estimators for the new model (1) and (17) are presented by using RKF without consideration of correlations and RKF with one-step correlated noises, respectively, based on the recurrence formula.
Algorithm 1: Approximate state estimators(using RKF) Step 1. predict Step 2. update where Next, take into account one-step correlated noises. The global optimal state estimators for dynamic system with onestep correlated noises are given in Pang et al. [32], according to which the approximate estimators are given in the following algorithm by treating remodeled system (1) and (17) as RKF with one-step correlated noises.
Algorithm 2: Approximate state estimators (using RKF with one-step correlated noises). Step 1. predict Step 2. update where and and R 1 is identical to (25).
Remark 3: Algorithm 2 provides the approximate state estimators for the linear dynamic system with one-step correlated noises. Similarly, the finite-step correlated noise approximate estimators can be proposed. However, this would bring heavy notations. Hence, the derivation process is omitted.

IV. N-STEP DELAY
Similarly, the new model under the assumption of n-step delay can be described with high-order Markov chain by applying the same method as section 3. Define m τ as the corresponding random variable of the observation at order τ as follow: Then {m i } is a high-order Markov chain corresponding to the n-step delay model. As for the order of {m i }, a theorem is given as follow. To prove ord {m i } ≥ 2n − 1, we only need to find one case where the probability distribution of current measurement relates to at least 2n − 1 previous steps. We pick out two branches of the multiple hypothesis tree of the n-step delay model independently (Fig. 1 is an example of 2-step delay model), as follows: In this case, previous 2n − 2 measurements(from y 2 to y 2n−1 ) are the same in Branch1 and Branch2 while the previous 2n − 1-th measurements(at y 1 ) are not. If we can deduce that the probability distributions of the current measurements(at y 2n ) are different, then ord {m i } ≥ 2n − 1.
Assume that the measurements of previous 2n − 1 steps in arbitrary two independent branches are the same, and the measurements at y m are unknown(m > 2n), as follows: Denote Z 1 as the measurement at y m in Branch3, and Z 2 as the measurement at y m in Branch4. Z 1 and Z 2 are random variables.
Regardless of the measurements at any step before y m−(2n−1) , when measurements from y m−(2n−1) to y m−1 (2n-1 steps) in Branch3 and Branch4 are the same, if we can deduce that Z 1 and Z 2 are i.i.d., then ord {m i } ≤ 2n − 1.
Assume that Z 1 and Z 2 are not i.i.d.. Then there must be one of the following four scenarios.
Given that the measurements of 2n − 1 steps before in Branch3 and Branch4 are the same, then in Branch4, ∃p ≥ m and q ≤ m − 2n s.t. y p = z q .
By Assumption 1, if the set of possible values of Z 1 and Z 2 is equal, then Z 1 and Z 2 have the same probability distribution. Note that Z 1 and Z 2 are not i.i.d. by our assumption. Without loss of generality, assume that P(Z 2 = z k 1 ) = 0. Thus k 1 = k 2 .
Since z k 1 , z k 2 ∈ {z m−n , z m−n+1 , · · · , z m−1 } and k 1 = k 2 , then k 1 > m − n or k 2 > m − n, say k 1 > m − n. z k 1 cannot be lost in Branch4, and the measurements of the previous 2n − 1 steps are the same in the two branches. Therefore, in Branch4, ∃p ≤ m − 2n, s.t. y p = z k 1 .
Note that |p − k 1 | > n, which is not possible in a n-step delay model, contradictory.
In Branch4, there is no such z l (l ≥ m) that came to the filter before y m . Otherwise, ∃k < m, s.t. P(Z 2 = z k ) > 0, contradictory.
In Branch3, ∃q ≥ m and y p ∈ {y m−n , y m−n+1 , · · · , y m−1 }, s.t. y p = z q , which is to say, there must be such z q (q ≥ m) that came to the filter before y m . Otherwise, P(Z 1 = z k 1 |∀k 1 < m) = 0, contradictory.
Thus, the 2n − 1 measurements before y m in Branch3 and Branch4 are not the same, contradictory to the assumption.
To sum up, Z 1 and Z 2 are i.i.d.. Therefore, ord {m i } ≤ 2n−1. Recall that we already proved ord {m i } ≥ 2n − 1, the order of With knowing the order of {m i } corresponding to the n-step delay model, {m i } can be transformed into a standard Markov chain through the method in section 2, denoted as {m i }. The same approach as in section 3 can be ported to n-step(n > 2) delay model unchanged.

V. NUMERICAL EXAMPLE
Consider the system of (1) and (2), the new estimators are applied in this section. Some common linear dynamic systems with 2-step random delay are given as simulation examples, and the results show that the new model proposed in this paper works better both in estimation accuracy and stability.
Tracking performance of an algorithm can be evaluated by estimating the square of Euclidean norm of the tracking error. To reduce errors that may arise from contingency, apply the Monte-Carlo approach for 100 runs, as follows: Example 1: Let the state transition matrix be   model is compared with the commonly used old model mentioned in section 2, and the traditional Kalman filter where delay is not taken into consideration. Under the settings of lower β 0 i and β 1 i , the probabilities of one-step delay and two-step delay are both relatively high. It can be seen from Fig. 4 that the curves of the average estimation errors of two algorithms in this paper are below the curves of the old delay model and traditional KF, which indicates that estimations given by the new model are more accurate. Moreover, the curve shape of the average estimation error of algorithms in this paper is smoother than that of the other two algorithms, which indicates that estimations given by the new model are more stable. It can be observed that the the old model even performs worse than traditional KF, which is because the settings of the old model are not reasonable, and this disadvantage is further magnified in the case of multiplestep delay. In the comparison of the two estimators proposed in this paper, the performance of the estimators obtained by using RKF directly and by considering one-step correlated noises is almost the same, the latter is very slightly better.
In Fig. 5, β 0 i = 0.6, β 1 i = 0.7, indicating that the probabilities of two-step delay and one-step delay are both relatively low. Similar to what Fig. 4 shows, the new filter proposed in this paper also has the advantages of more accurate and stable estimation, but the advantages are reduced to some extent. Thus, the new model performs best when the probability of delay is high. Several other sets of tests are also done by controlling variables, the results show that the change of β 0 i has much more effect than β 1 i on the estimation results.
Example 2: The dynamic system is modeled as (1) and (2). The state transition matrix  According to Fig. 6 and 7, the performance characteristics of each model are basically the same as those of the previous example. In general, the estimators of the new model are better than others in terms of accuracy and stability. However, when β 0 i and β 1 i are large, after 90 time points, RKF continued to work steadily, while the average estimation error of RKF with one-step correlated noises increases sharply with time, RKF with one-step correlated noises lose its stability since then. VOLUME 8, 2020  Fig. 4-Fig. 7, it can be seen that the average estimation error increases slowly as time steps increase. For a linear dynamic system with random delay, the estimation error will accumulate with the increase of steps, so it is natural that the tracking error will increase slowly. It is worth mentioning that we also run the simulation for far more than 100 steps, the result shows that the tracking error will increase very slowly as time steps increase. It can be expected that the estimation error will not be largely divergent.

VI. CONCLUSION
In this paper, the multiple-step random sensor delay of a general linear dynamic system was modeled more accurately and reasonably. The high-order Markov chain was applied to process the delay model so that it can be generalized as a mathematical model. Moreover, two approximate state estimators were proposed based on using RKF and RKF with one-step correlated noises, respectively. Some illustrative numerical examples were given to show that the new algorithm greatly outperforms those in former studies in both accuracy and stability. From a critical point of view, noises in the new model are infinitely correlated. Although RKF with one-step correlated noises has been tried in order to reduce the impact of infinite correlation, the results showed that it almost changed nothing, even worse in some cases, thus it is an issue to be investigated in future research.