Information Transfer With Respect to Relative Entropy in Multi-Dimensional Complex Dynamical Systems

In this paper, a rigorous formalism of information transfer with respect to relative entropy or Kullback-Leiber divergence within a multi-dimensional deterministic dynamical system is established. It is derived from the mechanism that the governance of the predictability change could come from the evolution itself and a transfer of the evolutions of multiple components for a given component. The presented formalism of three-dimensional systems and its several generalizations in high-dimensional systems provide a precise quantification of transfers among variables in complex dynamical systems, with which some properties are explored and given. These results of information transfers are different from that with respect to Shannon entropy in multi-dimensional systems, due to a minus sign which reflects the opposite notion of predictability vs. uncertainty. Explicit formulas are demonstrated and verified in the $\text{R}\ddot {o}$ ssler system and a four-dimensional system. These studies can be used to investigate the propagation of uncertainties and perform the dynamic sensitivity analysis statistically. The simulation results suggest that the generalized formalisms provide more underlying information about multi-dimensional dynamical systems compared with currently existing methods. It is beneficial for prediction and control of systems better, with broad application prospects in many fields.


I. INTRODUCTION
Uncertainty quantification is of great theoretical importance and practical significance in investigations of complex dynamical systems. It is very important to understand and predict their dynamics in analysis of interacting nonlinear systems. Causality analysis and information flow/transfer are closely related concepts, whereby information or knowledge of certain states can be thought of as coupling influence onto the future states of other processes in a complex system [1]. This flow or transfer related to the spatiotemporal structure of predictability is important as it can determine how predictability in one place is altered due to other places and how uncertainties propagate in the system, which helps to identify the source regions of unpredictability [2].
The associate editor coordinating the review of this manuscript and approving it for publication was Byung-Gyu Kim .
Transfer and transferlike entropies have been, and still are, the subject of much research on both theoretical and practical shortcomings, and the very popular techniques for studying causality in coupled dynamics shown according to these continued efforts are information-theoretic tools [3]. A great many works exist in the literature. As such, the applications of Schreiber's transfer entropy [4], [5], the measurements of information transfer in different disciplines [6]- [9], the measurement uncertainty of hypertension and stress on kidney [10], the comparison of Shannon, Kullback-Leibler and renormalized entropies within successive bifurcations [11], the connection between topological and algebraic entropy [12], the analysis of heart rate variability using fuzzy measure entropy [13], the measurement entropy of semi-independent hyper MV-algebra dynamical systems [14], the information entropy of hyper MV-algebras [15], the Algebraic and Shannon entropies of commutative hypergroups [16], the evaluation of an information system effect on the learning of the space structure [17], the investigation of the ergodic properties of probability dynamical systems using concept of entropy [18], etc. Given the broad and pressing need to realistic applications on complex dynamical systems, information flow can be used substantially. During the past decades, the formalisms of information transfer have been established empirically or half-empirically based on observations of diverse disciplines, for example, time-delayed mutual information [8] and transfer entropy [4]. Until 2005, Liang and Kleeman put forward a rigorous formalism to measure information transfer of two-dimensional (2D) systems with the dynamics are given, which can be referred as LK2005 formalism henceforth [19]. The idea is derived from interactions between two components in complex dynamical systems. In terms of its applications, the LK2005 formalism has yielded remarkable results by thoroughly describing the statistical behavior of a system. However, the LK2005 formalism and its generalized forms [20]- [23] are only suitable for some dynamical systems of arbitrary dimensionality with successful applications between two variables, but it is invalid to quantify uncertainty of many real-world coupled systems. Considering realistic applications of sensitivity analysis of an aircraft system with interactions between multiple components, recently, a rigorous formalism of information transfer with respect to Shannon entropy among the components in multi-dimensional deterministic complex dynamical systems has been established in [24] based on the LK2005 formalism to deal with the uncertainty change of one component given by the other components. The measurements of information transfer can help us to better understand the intrinsic mechanism and explore deeply hidden information in the evolution of multi-dimensional dynamical systems. The above mentioned formalisms of information transfer are given with respect to shannon entropy or absolute entropy, which are the measurement of uncertainty.
Causation inference and information flow are classical topics in diverse disciplines nowadays, which remain advancements in continuing to strengthen the theory and push the applications [1]. There is another measure of the predictability in information theory, that is, relative entropy, which is also called Kullback-Leibler divergence. The Kullback-Leibler divergence was introduced as the discrimination information between two distributions [25]. The Kullback-Leibler divergence is a measure of how one probability distribution is different from a reference probability distribution [26], whose advantages have been shown in that it possesses some appealing properties compared to Shannon entropy [27]. It is a measure on how much additional information is added when the reference probability is fixed as the initial distribution and for dynamical systems with the joint density evolve [28]. Furthermore, it has the invariance upon nonlinear transformation and the consistency with the second law of thermodynamics in the context of Markov chain [29]- [31]. Therefore, it is well accepted as a measure of predictability [29], [32]. Since the information flow is an important physical problem in practical concern and nature, a formalism of information flow with respect to relative entropy has been examined in 2D deterministic system [2] to study how predictability propagates in physical space better. However, it cannot capture the indirect information transfer, that is, it fails to identify whether the information is from a direct transfer via the other variable or an indirect transfer via some other variables [33], [34].
Hence, we develop a formalism of information transfer with respect to relative entropy to several variables of multi-dimensional dynamical systems for catering to practical requirements such as sensitivity analysis of aircraft systems in this paper. In other words, we extend the results in [19] to the information flow with respect to relative entropy between groups of components, rather than individual components in arbitrary multi-dimensional dynamical systems when dynamics is fully known. It is a follow-up of the study in [2], [24]. In addition, the relationships among several generalized formalisms are highlighted. Compared with the previous formalisms, the generalized formalisms can be used to quantify information transfer from several variables and high order interactions among them to another variable. Furthermore, the generalized formalisms can help to identify direct or indirect information transfer between variables in dynamical systems. It is beneficial for exploring the complexity of evolution of multi-dimensional dynamical systems. Meanwhile, it is also a measure of performing dynamic sensitivity analysis. Considering the large body of aircraft designs work on uncertainty studies, this research is expected to provide some help to determine design parameters and guide decision-making for predicting complex computer models. It is significant to investigate real-world problems.
The structure of this paper is as follows: Section 2 recalls the recent developed formalisms of information flow in multi-dimensional systems briefly; In Section 3, multi-dimensional formulas of information transfer with respect to relative entropy are established. Details on the derivations of the formalisms and the related properties are demonstrated; Section 4 gives a description about the formalisms with multi-dimensional applications; the summary of this paper is given in Section 5.

II. MULTI-DIMENSIONAL FORMALISM OF INFORMATION TRANSFER WITH RESPECT TO SHANNON (ABSOLUTE) ENTROPY
For a three-dimensional (3D) continuous and deterministic autonomous system, where F = (F 1 , F 2 , F 3 ) is a known flow vector with F i = F i (x 1 , x 2 , x 3 ) for any i = 1, 2, 3 and x = (x 1 , x 2 , x 3 ) ∈ for which the sample space is assumed to be a direct product of the random variables corresponding to the sample values (x 1 , x 2 , x 3 ). For convenience, we will use the notation ρ instead of the notation ρ(x 1 , x 2 , x 3 , t) except where noted. The probability density ρ associated with (1) satisfies the Liouville equation [35]: After some algebraic manipulations about (2) with assuming that ρ vanishes at the boundaries (the compact support assumption for ρ and the assumption is reasonable in real-world problems [19]), it is found that the time rate of the joint absolute entropy change of X 1 , X 2 and X 3 , where E(∇ · F) = ρ(∇ · F)dx 1 dx 2 dx 3 . The property holds for deterministic systems of arbitrary dimensionality [20]. A detailed derivation on (3) is referred to [21].
As mentioned above, the time rate of change of H equals to the mathematical expectation of the divergence of the flow vector F. The marginal density of x k in 3D systems is where i, j, k = 1, 2, 3 with different i, j, k at the same time. The shannon entropy for the component Equation (4) states how H k evolves with time. The evolutionary mechanism of H k derives from two parts: One is from the evolution itself, dH * k dt ; another from the transfers of X i and X j according to the coupling in the joint density distribution ρ. When X k evolves on its own, denoted by where E means the mathmatical expectation with respect to ρ. The rate of information flow/transfer with respect to absolute entropy from X i , X j to X k is where ρ(x k ,t) . Moreover, when several variables are involved, the information transfers from components X i , X j , . . . , X n to X k of multi-dimensional continuous and deterministic autonomous systems has the following form: Equation (7) can be used to quantify the information transfers from several variables and high order interactions among them to the other variable. In addition, the generalized formalism can be reduced to 2D cases when only considering information transfer between two variables of a system.

III. MULTI-DIMENSIONAL FORMALISMS OF INFORMATION TRANSFER WITH RESPECT TO RELATIVE ENTROPY A. THE FORMALISMS OF 3-DIMENSIONAL SYSTEMS
At first, we consider a three-dimensional dynamical system (1): We research the predictability change of one component of x contributed by the other two components, that is to say, how information is transferred from two variables to the other variable with respect to relative entropy. It is known that the definition of relative entropy, for two joint probability densities ρ and q of x, It is a measure of the distance between ρ and q. In general, q is the initial density or equilibrium density. In this paper, it can be treated as the initial density except where specified. Now consider the information transfer from X 2 and X 3 to X 1 . The marginal relative entropy of X 1 is where

Equation (8) can be written as
Derivatives for (10) with respect to t and combining (2) and (3), the time rate of the joint relative entropy change of X 1 , X 2 and X 3 is 39466 VOLUME 8, 2020 The evolution equation of ρ 1 is derived by taking the integral of (2) with respect to x 2 and x 3 over the subspace 2 × 3 : After multiplying by (1+log ρ 1 ) for (12) with some algebraic manipulations: By integrating for (13) and using the assumption that ρ vanishes at the boundaries, the evolution equation of marginal absolute entropy H 1 is Derivatives for (9) with respect to t, then substituting (12) and (14) with the fact ∂(F 1 ρ) ∂x 1 dx 1 dx 2 dx 3 = 0 by the compact support assumption for ρ, similar to the two-dimension situation in [2], the evolution of the marginal relative entropy here q 1 does not depend on time. For convenience, dD 1 KL dt is written as dD 1 dt in the following context. Similar to the situation with respect to absolute entropy, the evolution of the predictability of X 1 , dD 1 dt derives from two parts: one is from the evolution of itself, dD * 1 dt ; another from the influences of X 2 and X 3 according to the coupling in the joint density distribution ρ. The latter is the information flows from X 2 and X 3 to X 1 , denoted by T D 2,3→1 . From (11), the time change of D only depends on ∇ · F and F · ∇ log q. If it evolves on its own, one may argue that the entropy with X 1 changes with ∂F 1 ∂x 1 and F 1 · ∇ log q 1 based on the concept of joint relative entropy. According to (11), it should have an equation: It is worth note that (16) can be derived from (51) with n = 3. Similar to (35) in [2], we obtain the following theorem: Theorem 1: For system (1), the information flow with respect to relative entropy from X 2 and X 3 to X 1 is The detailed derivations of Theorem 1 are demonstrated in Appendix A. As in Section II, for any other transfers, i.e., T D i,j→k , we may insert 1,2 and 3 correspondingly to obtain all. There is a clear interpretation for information flow/transfer associated with the formulas (17). As stated in [2], the information transfer from X 2 and X 3 to X 1 , T 2,3→1 quantifies the influence of X 2 and X 3 on the predictability of X 1 . Meantime, when T 2,3→1 > 0, it means that the evolution of X 2 and X 3 favors the prediction of X 1 , that is to say, it will make X 1 more predictable. Otherwise, when T 2,3→1 < 0, it implies that the evolution of X 2 and X 3 reduces the predictability of X 1 , which means that it will bring to X 1 more uncertainties. The positive or negative information transfer T 2,3→1 means that the evolution of X 2 and X 3 gain or reduce the uncertainty of X 1 . Meantime, It is found that the formula (6) is same as the formula (17), except for a minus sign. It conforms with the concept of shannon entropy and relative entropy as a measure of uncertainty and predictability, respectively.

B. THE FORMALISMS OF n-dimensional systems
Similarly, for n-dimensional systems with n > 3 we can obtain the formalism of information flows with respect to relative entropy from X 2 , X 3 , . . . , X n to X 1 using the same derivation with the Liouville equation that is we can insert 1, 2, . . . , n to (20) correspondingly to obtain all transfers. The obtained formalism is the transfer to one component from all other components in multi-dimensional dynamical systems. Now one can consider how to screen out the part contributed from x 2 and x 3 to x 1 from (20). The difficulty is how to obtain the evolution of D 1 with the influence of x 2 and x 3 excluded, which is different from the VOLUME 8, 2020 In fact, we have known that the evolution of D 1 includes the transfer from X 2 and X 3 and the evolution without effect from X 2 and X 3 , which can be written as dD 12 \3 \ dt in systems with dimensionality higher than 3. It is the tendency of D 1 with x 2 and x 3 fixed at time t and the subscript 2 \3 \ signifies the effect of x 2 and x 3 is removed. Obviously, when ρ involved with the subscript 2 \3 \, ρ 2 \3 \ , it signifies marginalization of the component, that is, Without loss of generality, we consider the information transfer with respect to relative entropy from X 2 and X 3 to X 1 in multi-dimensional systems firstly, which is the difference between dD 1 dt and Now the main problem is converted to compute dD 12 \3 \ dt . It cannot be obtained via using the Liouville equation (19) corresponding to multi-dimensional deterministic dynamical systems, as the dynamics is changed with time t on manipulating x 2 and x 3 [2]. Based on the previous derivation by Liang [21], the evolution of D 1 when X 2 and X 3 is fixed, dD 12 \3 \ dt in multi-dimensional systems can be written as Here we compute it by using the definition of derivatives and approach it through to find how the relative entropy D 1 increases from time t to time t + t with the limit t → 0 in a discretization of the system (18). The discrete mapping form of system (18), : −→ is which is an approximation of system (18) up to the first order of t and drives . To avoid confusion and express shortly, x(t + t) = (x(t)) can be replaced by y = (y 1 , y 2 , . . . , y n ) in this paper. The above mapping is as follows in component form: Since x is steered forth with the transformation = ( 1 , 2 , . . . , n ) from time step t to t + 1, correspondingly its density ρ is driven forward by Frobenius-Perron operator (F-P operator) [35]: P : with any subset of , ω. Liang [2], [21] showed some useful properties of and its corresponding F-P operator. Firstly, and its variables are invertible as t → 0 with the form the high order terms may be omitted with t → 0. Secondly, the Jacobian matrix of and its inverse are and which can be learned from . The last property is that the corresponding F-P operator of the invertible is written clearly as We can fix x 2 and x 3 to obtain the influence from X 2 and X 3 excluded, correspondingly the mapping is modified as follows: 2 \3 \ : . . . . . .
with x 2 and x 3 fixed as parameters at time t and an associated modified F-P operator, P 2 \3 \ . The marginal density of x 1 at t + t with x 2 and x 3 fixed as parameters at t, . . , y n )dy 4 . . . dy n is dependent on x 2 and x 3 as well as has extra dependencies about x 4 , x 5 , . . . , x n introduced by the conditional density of x 2 and x 3 on x 1 , x 4 , . . . , x n [23]. It is worth noting that, we will use 4...n instead of 4 × · · · × n throughout for notational convenience. Therefore, the marginal absolute entropy of the first variable evolved from H 1 with contribution from X 2 and X 3 excluded as t → t + t is The marginal relative entropy of the first variable evolved from D 1 with contribution from X 2 and X 3 excluded as t → t + t can be obtained in the same way, that is, (32) here ρ 4...n is the joint probability density function of (x 4 , x 5 , . . . , x n ). For n-dimensional system (18) with dynamics given, the time rate of the marginal relative entropy change on X 1 with the influence of X 2 and X 3 excluded is here See Appendix B for the detailed derivations of (33).
If we replace X 2 , X 3 with a variable Z of (35) in [2], it seems that it is possible to get the information transfer from (X 2 , X 3 ) to X 1 , however, it is a misleading problem. Firstly, if we replace X 2 , X 3 by a variable Z , the two equations F 2 and F 3 are lost, which resulting in the incorrect information flow. As we can see from equation (19), all the equations are needed to get the probability densities. Secondly, if the given form is taken, we lost the information between other variables and X 2 , X 3 , as well as the indirect information transfers associated with X 2 and X 3 . Finally, (34) is a more universal form, while the modified form is a special form which can be valid only at some special conditions.
We can also insert i, j and k correspondingly to (34) to obtain other transfers from X i and X j to X k , T D i,j→k and all information transfers from X i , X j , . . . , X n to X k following the same procedure: and The information flows obtained by (35) and (36) are asymmetric, for instance, while X k gains information from X i or X j , or X i and X j , X i or X j might have no dependence on X k in system (18). A property of information flow is its asymmetry among the components, which implies causality [36]. In addition, when F k is independent of x i and x j , there should be no information transfer from X i and X j to X k . It is consistent with the information flows defined in (35). As a matter of fact, another property of the information flow is given below as an extension of (36) in [2].
Theorem 3: For system (18), if F k is independent of x i and x j with different i, j and k, then the information flow with respect to relative entropy from X i and X j to X k , T D i,j→k = 0. Proof of Theorem 3: According to the formalism (35) of information transfer for system (18) with the notation of F k = F k (x k ), after integrating with respect to x i and x j for ρ, θ i,j|k and i,j|k , respectively, where the notation i \ signifies that the component i is excluded from the set of variables in systems.
Then, integrating for (35) with respect to x i and x j , and using the assumption that ρ is compactly supported, However, there is possibility that the flows in other directions may be nonzero when F i depends on x j and x k or F j depends on x i and x k . As a special case of (35), (17) is also equal to zero with the case of F 1 is independent of x 2 and x 3 , that is, Likewise, Theorem 3 can be generalized to multi-dimensional cases associated with (36) in the same way. Moreover, the information flow with respect to relative entropy is invariant upon coordinate transformation.
Proof of Theorem 4: Since we consider the information flow with respect to relative entropy from X 2 and X 3 to X 1 , the transformation cannot be made for x 1 , x 2 and x 3 . Assuming that the transformation G : x → v(R n → R n ) has the following form: then system (18) can be changed to Actually, here the Jacobian of G = (G 1 , G 2 , . . . , G n ), J is the same as that of (G 4 , G 5 , . . . , G n ), J 1 \2 \3 \ . And the density of v is as follws: ρ(v) = ρ(x)|J −1 1 \2 \3 \ | according to the Frobenius-Perron operator result. The information flow with respect to relative entropy from v 2 and v 3 . In addition, it is interesting to note that if the evolution of X 3 is independent of X 1 and X 2 , the formula (35) in [2] is changed to it is consistent with the formula (34).

IV. APPLICATIONS: THE ROSSLER SYSTEM AND ROSSLER HYPERCHAOS SYSTEM A. THE rössler system
In this subsection, we present an application study of the information flows with respect to relative entropy about the Rössler system [37]: where a, b and c are parameters, x 1 , x 2 and x 3 are the system state variables, and t is time. The chaotic attractor of Rössler system with a = 0.2, b = 0.2, c = 5.7 is shown in Figure 1. To compute the information flows with respect to relative entropy within three variables through formula (17) of 3D systems, the key step is to obtain the joint probability density function ρ(x 1 , x 2 , x 3 ) of X . Actually, the evolution of the joint density ρ(x 1 , x 2 , x 3 ) in deterministic systems with known dynamics can be obtained by solving the Liouville equation but the computational load. Here we estimate the joint density ρ(x 1 , x 2 , x 3 ) by the way of counting the bins according to the ensemble prediction of the Rössler system at each time step.
The Rössler system is solved by applying a fourth order Runge-Kutta method with a time step t = 0.01 to generate the ensemble. According to Figure 1, the computation domain is restricted to ≡ [− 16,16] 4,28], for including the attractor of the Rössler system. We discretize the sample space into 320 × 320 × 320 = 32768000 bins such that the attractor lies within the computation domain and one draw per bin on average via making 32768000 random draws. We assume X is distributed as a Gaussian process N (u(t), Σ(t)), with a mean u and a covariance matrix Σ: in the initial conditions. After using different parameters u k and σ 2 k (k = 1, 2, 3) to compute information flows for the Rössler system, we find that the final results are the same and the trends stay invariant. It is open for different experiments to adjust the parameters u k and σ 2 k . Here we only show the results of one experiment with u 1 = 8, u 2 = 2, u 3 = 10 and σ 2 k = 4. The ensemble is driven by drawing sample randomly with a pre-established distribution ρ 0 (x). We obtain an ensemble of X and estimate the three-variable joint probability density function ρ(x 1 , x 2 , x 3 , t) of the Rössler system through counting the bins at every time step. As these equations of the Rössler system are integrated forward, ρ can be estimated at each time step with describing the statistics of the system. The detailed discussions on probability estimation through bin counting can be seen in [21], [38]. An estimated marginal density corresponding to x 1 , x 2 and x 3 of the Rössler system are displayed in Figure 2. The information transfer with respect to relative entropy related to the couple effect from two variables to another variable in 3D systems can be computed through formula (17). Firstly, we estimate the transfers T D i,j→k , i, j, k = 1, 2, 3 with different i, j, k at the same time. A nonzero T D i,j→k means that how much predictability that X i and X j bring to X k . The calculation results are plotted in Figure 3, whose relative values represent the magnitudes of information transfer. When we consider the linear system with only two equations F 1 and F 2 in the Rössler system, T D 2,3→1 is larger than T D 1,3→2 according to the magnitude of parameters and the definition of the generalized formalisms. However, the transfer from X 1 and X 3 to X 2 is smaller than the transfer from X 2 and X 3 to X 1 in Figure 3. It is consistent with the characteristics of complex systems [38], that is to say, emergence does not result from rules only [39]- [41]. The simulation results show the efficiency of the generalized formalisms by revealing the inherent relation in complex systems simultaneously. Moreover, as shown in Figure 3, any two variables reduce the predictability of the other variable and all information flows go to constants over time, which means that the system reaches a steady state gradually. The positive or negative information transfers imply that the two components may increase or decrease the predictability of another component. Repeated experiments with different initialization are in line with the above results.
For the sake of demonstrating a more intuitive comparison and revealing some underlying information in the chaotic dynamical system better, we also give information transfer between two components, T D 1→2 through using the formula (35) in [2], then compare T D 1→2 with the transfer T D 1,3→2 in Figure 4. According to the equations of Rössler system, it is clearly shown that T D 3→2 must vanish by the principle of nil causality [23], that is T D 3→2 = 0. It seems that the transfer T D 1→2 should be equal to T D 1,3→2 in Rössler system. What makes the results interesting is that there is a difference between the graph T D 1→2 and T D 1,3→2 in Figure 4. In fact, it is well-founded. As mentioned in [2], indirect information transfers could take place through a third party or more parties due to the fact that two variables could be related in a high-dimensional system. More specially, indirect information transfers mean that they may occur in multi-dimensional dynamical systems, though one variable does not affect the dynamics of the second variable directly, it does so via another variable. In other words, x i affects x j the dynamics of which in turn affects x k in dynamical systems [34]. There is no information transfer from x 3 to x 2 in the Rössler system, however, T D 3→1 and T D 1→2 do not vanish, hence information does transfer from x 3 to x 2 with the help of x 1 indirectly. Indirect information transfers can help to identify which variable leads to an indirect non-zero transfer.
From the above example, we can find that there exists hidden sensitivity information by computing the information transfer of high-dimensional dynamical systems: we can identify the direct and indirect componentwise relations and understand the indirect way of information flows. It will benefit many practical fields such as sensitivity analysis among the variables in dynamical systems. Meanwhile, some underlying information can be revealed through quantifying transfers among the variables of the chaotic dynamical system.
In addition, we plot the transfers, T D 2→1 , T D 3→1 and T D 2,3→1 in Figure 5, then compare the magnitudes of three flows. The absolute value of the transfer measures information flows among the variables [23]. We can conclude that x 2 is more sensitive to x 1 than x 3 to x 1 according to the measurements of information transfers. Furthermore, T D 1→2 in Figure 4 is different from T D 2→1 in Figure 5, implying the property of asymmetry of information transfer.

B. A FOUR-DIMENSIONAL DYNAMICAL SYSTEM
To show that the generalized formalisms can be used to multi-dimensional dynamical systems efficiently, we consider the following four-dimensional(4D) system:   (1, 1, 1, 1). These trajectories are 3-dimensional plots of here x i (i = 1, 2, 3, 4) make up the system components and the parameters are a = 12, b = 23, c = 2.1, d = 6, e = 0.2. A computed attractor of the 4D system with initial value (1,1,1,1) is shown in Figure 6. Follwing the above procedures, the joint density ρ( which includes an attractor of the 4D system can be selected to estimate the four-variable joint probability density function. The following computation is demonstrated by applying a fourth order Runge-Kutta method. Similarly, we only show the results of one experiment after computing information flows multiple times by using different parameters. Suppose that X is distributed as a Gaussian process N (u(t), Σ(t)), with a mean u and a covariance matrix Σ in the initial state:

   
We discretize the sample space into = 100 × 100 × 100 × 100 = 100000000 bins to ensure it is enough to understand the information transfer and the evolution of the system. An estimation result of four marginal densities is shown in Figure 7. Similarly, we discuss the coupling effect from three components to the other component, T D 2,3,4→1 , T D 1,3,4→2 , T D 1,2,4→3 and T D 1,2,3→4 by using (20) to compute the information transfers within the 4D system at first. The resulting transfers are shown in Figure 8. We can quantify the predictability and give the dynamical influence among variables of a system according to the simulation results.
Secondly, we compute the transfers from any variables to the other variable in the 4D system through (36). There are twelve transfers on the couple effect from two components to the other component in this system, here we only give the VOLUME 8, 2020   Figure 9. Any two other variables provide the predictive impact on x 1 from Figure 9. Besides, the results show that x 3 and x 4 increase the predictability of x 1 , but the other two pairs decrease the predictability of x 1 .
In particular, we plot the transfers from any one variable and the high order interactions among them to x 4 in Figure 10. We can find that x 2 is more sensitive to x 4 than x 1 to x 4 and x 3 to x 4 according to the resulting transfers.
Using the same procedures, we can rank the importance of variables in an aircraft system. The uncertainty analysis and sensitivity analysis process of an aircraft system with respect to variables, parameters and uncertainty factors is one of the key steps for determining the optimal search direction and guiding the design and decision-making. It aims at predicting complex computer models by quantifying the sensitivity information of the coupling variables and leads to high performance aircraft designs. We can use the generated formalisms of information flow as a sensitivity analysis index to perform dynamic sensitivity information analysis due to the fact that Liouville equations and Frobenius-Perron analysis describe an ensemble of trajectories. The uncertainty can be quantified among the variables, which benefits us to improve understanding of dynamical systems.

V. CONCLUSION
In this paper, we develop a method to measure information transfers with respect to relative entropy among multi-dimensional complex dynamical system components based on the previous formalisms [2], [24]. Information transfers from some components to another component are quantified via the rigorous and general formalisms and some related theoretical properties are given. The proposed formalisms provide a measure of predictive impact and information flows among variables, which enable us to better understand the physical mechanism underlying the superficial behavior and explore deeply the hidden information in the evolution of complex dynamical systems. With contrast to the 2D formalism, the generated formalisms have advantages to reveal much more information on multiple cases. We quantify not only the correlation among variables and the system behavior, but also how much they influence between each other. Furthermore, our formulas are capable to identify the direct transfer or indirect transfer from some variables to another variable, and also can quantify the high order interactions among variables. It benefits us for advancing our understanding of coupled nonlinear dynamics.
The uncertainty can be quantified among the variables of aircraft systems by the generalized formalism presented in this paper, which benefits to reveal the nonlinear relationships and to assess the dynamic stability and predict the forthcoming states of aircraft designs. Since the formalism is built on the statistical nature of information, it could be used to perform dynamic sensitivity analysis in multi-dimensional complex systems. This issue makes it possible not only to quantify the amount of uncertainty created by information transfers among variables of a system, but also to understand how these transfers influence the system behavior. Moreover, it also assists with determination of sensitive parameters in complex dynamical systems. Thus, it may lead to improvement in estimation, prediction and control tasks in aircraft systems. In practice, it is not easy to give the dynamics analytically for complex multi-dimensional dynamical systems. Considering the fact that many critical data-driven problems take advantage of the progress in data-driven discovery of dynamics [33], [34], [42], further investigations can be conducted into the dynamic-free formulation to analyze information transfers of multi-dimensional systems and to make a comparison between the results of current work and that of the computations on information transfer measure from time-series data [43]- [46]. Moreover, the formalisms could be further generalized to multi-dimensional stochastic dynamical systems and time-delay systems in future work. In addition, future study also includes that how the information flow can be deployed in the frame of dynamic sensitivity analysis as a novel indicator.

APPENDIXES APPENDIX A THE PROOF OF THEOREM 1
Since we can obtain the formalism of information transfer with respect to relative entropy from X 2 and X 3 to X 1 with the compact support assumption for ρ: where ρ 2,3|1 (x 2 , x 3 |x 1 ) = ρ ρ 1 .