Stability and Synchronization of Switched Multi-Rate Recurrent Neural Networks

Several designs of recurrent neural networks have been proposed in the literature involving different clock times. However, the stability and synchronization of this kind of system have not been studied. In this paper, we consider that each neuron or group of neurons of a switched recurrent neural network can have a different sampling period for its activation, which we call switched multi-rate recurrent neural networks, and we propose a dynamical model to describe it. Through Lyapunov methods, sufficient conditions are provided to guarantee the exponential stability of the network. Additionally, these results are extended to the synchronization problem of two identical networks, understanding the synchronization as the agreement of both of them in time. Numerical simulations are presented to validate the theoretical results. The proposed method might help to design more efficient and less computationally demanding neural networks.


I. INTRODUCTION
Recurrent neural networks (RNNs) were developed in the 1980s [1]- [3]. In a RNN, each neuron is represented by a node of the network and it is directly connected to the rest of the neurons. RNNs have been extensively studied in the literature since their multiple applications in different fields, such as signal processing [4], automatic control [5], [6], financial applications [7], [8] and so. Such applications heavily depend on the dynamic behavior of the networks [9]. Therefore, the analysis of these dynamic behaviors is a necessary step for the practical design of recurrent neural networks. In this regard, excellent results have been produced in the recent years, especially for RNNs with timedelays [10]- [12], but also in the context of event-triggered transmission of information [13]- [15], discontinuous activation [16], [17], etc.
In this work, we focus on a particular case of RNNs that we call multi-rate recurrent neural networks (MRRNNs). In general, a multi-rate system is a discrete system where the different elements have different sampling periods, but The associate editor coordinating the review of this manuscript and approving it for publication was Qiuye Sun . which, in general, satisfy that a meta-period exists such that each period is repeated an integer multiple of times in that meta-period. This has been especially used in automatic control [18]- [20], for example, to build a closed-loop system where the input is updated several times with only one output measurement. However, it has proved to be applicable in many other fields such as disk drive servo systems [21], [22], chemical analyzers [23], [24] or visual feedback [25], [26]. In this regard, several approaches have been considered for RNNs which involve different time scales, such as clockwork RNNs [27], where the time steps are arbitrary pre-fixed, hierarchical multiscale RNNs [28], which learn update probabilities for different units using the straight-through estimator, or dilated long-short term memory (dLSTM) [29], where the time steps are obtained from all the available training data. This kind of networks might be useful in many scenarios since training and evaluation time can be reduced [27]. Additionally, some experimental applications have been investigated, like video-based pedestrian identification [30], [31]. In spite of that, to the best of the authors' knowledge, analyses of the dynamic behaviors of the networks have not been performed in the literature. This kind of analysis can provide tools for the efficient design of the networks. See for example, [32], where it is shown that a dual-rate strategy in the path following of a mobile robot can provide better results than an inferential strategy when losses of information are reported, and other scenarios such as power systems [33], [34] or pipeline networks [35], where the application of the method may provide benefits.
To generalize the model, we consider also a practical aspect that appears normally in neural networks [36]- [38]: We assume that the studied RNNs may exhibit a switched behavior, i.e. the state feedback and the weight connection matrices of the RNN are not fixed and may vary among different values with a switching signal. Switched systems are a special kind of hybrid systems with applications in many fields such as mechanical systems, power systems automotive industry, etc. [39].
We study the multi-rate approach from two complementary points of view. First, we study the stability of a switched MRRNN, which we define as a network where each neuron (or group of neurons) has its activation period, and therefore, the information of each network is asynchronously transmitted to the other neurons. Second, we study the synchronization problem of two identical switched MRRNs, understanding the synchronization as the agreement of two different processes in time. That is, we design the appropriate feedback law such that two identical switched MRRNNs with different initial conditions are controlled to reduce the difference between their states. In summary, the main contributions of the paper are i) the design of a general dynamical model for switched RNNs taking into account different sampling periods for each neuron or group of neurons, ii) the development of a method to guarantee the exponential stability of this kind of system, and iii) the extension of the method to guarantee the exponential synchronization of two identical networks.
The remainder of the paper is organized as follows. Section 2 introduces the preliminary concepts and results used in the paper. In Section 3, we describe the model of the switched MRRNN. Section 4 is devoted to the stability analysis of switched MRRNNs. In section 5, the synchronization analysis of switched MRRNNs is carried out. In Section 6, the theoretical results are applied to numerical examples to validate the analyses. Finally, concluding remarks are provided in Section 7.

II. PRELIMINARIES
The n-dimensional real space is denoted by R n , R >0 is the set {x ∈ R|x > 0}, and R ≥0 = R >0 ∪ {0}. Similarly, the n-dimensional rational space is denoted by Q n , Q >0 is the set {x ∈ Q|x > 0}, and Q ≥0 = Q >0 ∪ {0}. We refer to the euclidean norm of vector x ∈ R n as x := √ x T x. Let A ∈ R n×m , the transpose matrix of A is denoted by A . We denote the identity matrix of dimensions n × n by I n and the zero matrix of dimensions n × m by 0 n×m . Symmetric matrices of the form A B B C are denoted as A B C . We further denote a symmetric positive-definite matrix P ∈ R n×n as P 0. Matrices P 0, P ≺ 0 and P 0 refer to symmetric positive-semidefinite, negative-definite, and negativesemidefinite matrices, respectively. For a matrix A ∈ R n×n , we denote by λ max (A) and λ min (A) to the maximum and minimum eigenvalues of A, respectively.
Definition 1 [40]: The equilibrium point x of a discretetime process x(k+1)=f(x(k)) is said to be exponentially stable if there exists constants 0 < ρ < 1, c 1 ≥ 0 and c 2 We define ceil(a) as the function that rounds the real number a to the nearest integer greater than or equal to a and α : N ∪ {0} → N as a piecewise constant function called switching signal, which takes values in a finite set N . Then, we define the average dwell time as follows.

III. SYSTEM DESCRIPTION
Consider a n-neuron neural network, where the state of each neuron is x i ∈ R, and each of them is activated with period h i ∈ Q >0 . Therefore, a meta-period h exists such that n i h/h i is an integer for all i, as shown in Figure 1. Definex i (t) as the last activation of the neuron i, i.e.
So, the evolution of the neuron i is defined by the following difference equation for i = 1, . . . , n, k i ∈ N, and where 0 ≤ c i < 1 is the state feedback coefficient of neuron i, A i ∈ R 1×n is the weight connection matrix of neuron i, g(x) [g 1 (x 1 ), g 2 (x 2 ), · · · , g n (x n ))] is the neuron activation vector function, and J i is a constant external input vector. In the next, we consider that the neuron activation vector function satisfies the following assumptions.
• A1. The neuron activation vector function g(x) is bounded and satisfies g(0 n ) = 0 n .

IV. STABILITY ANALYSIS
Once the switched MRRNN has been modeled as a switched discrete-time system (7) with a meta-period h in which each neuron is activated an integer number of times including the beginning and end of the meta-period, it is possible to develop the conditions to guarantee its exponential stability through Lyapunov methods [41], [42] and to compute the average dwell time for the switching signal. These conditions are established in the following theorem.

V. SYNCHRONIZATION ANALYSIS
In this section, we consider the exponential synchronization of two identical switched MRRNNs. As aforementioned, we understand the synchronization of two systems as the agreement of both of them in time. This means that both switched MRRNNs should exponentially converge to the same temporal evolution of their respective states. To do that, we define a drive system which should be followed by a response system by means of a control input appropriately designed. First, let us consider (7) as the drive system of the synchronization problem. The neuron i of the response system has the form with u i to be defined later. Defining the control input as in (18) allows us to combine (17) for all neurons to obtain the response system ψ((k + 1)h) = C(α(k))ψ(kh) for k ∈ N, where ψ(k) = [ψ 1 (k), ψ 2 (k), · · · , ψ n (k)] ∈ R n is the neuron state vector of the response system,ˆ (k) [ψ(kh+t 1 ),ψ(kh+t 2 ), · · · ,ψ(kh+t m )] ,ψ(kh+t i ) is the vector of last activations for the response system, F(ˆ (k)) [f (ψ(kh + t 1 )), f (ψ(kh + t 2 )), · · · , f (ψ(kh + t m ))] and U (kh) = [u 1 (kh), u 2 (kh), · · · , u n (kh)] ∈ R n is the control input.
To design the control input, we define e(k) ξ (k) − ψ(k) and H (ˆ (k),ˆ (k)) F(ˆ (k)) − F(ˆ (k))). Note that from (5), it can be verified that h i (0, 0) = and ∀e i (k) = 0 for i = 1, 2, . . . , n. Therefore, we define the control input as where K ∈ R n×n is a switching gain matrix to be determined. Finally, we can write the closed-loop error dynamics as e(k + 1) = (C(α(k)) − K (α(k))) + A(α(k))H (ˆ (k),ˆ (k)). (22) Now, if the closed-loop error dynamics defined are proved to be exponentially stable, then it is also proved that drive and response systems exponentially converge, and therefore, that they are exponentially synchronized. This result is stated in the following theorem.
Moreover, the feedback gains K (α(k)) of the response system (19) can be obtained as follows.

VI. NUMERICAL EXAMPLES
In this Section, several examples are described to validity the analyses developed in Section IV and V. For that, let us recall Example 1: Simulation 1 (Stability Analysis): Consider the MRRNN in (7) with N = {1, 2}, h 1 = h/2, h 2 = h/3 as described in Example 1, and where and an activation function g i (s) = tanh(s), such that + = diag{1, 1}, − = diag{0, 0}. Next, we can use Theorem 1 to check the stability of the system, which depends on parameters β and µ. In this case, it can be proved that LMIs (8)- (9) are satisfied for β = 0.8 and µ = 1. This implies that (8) Since µ = 1 (and consequently P(1) = P(2)), the MRRNN is exponentially stable for any arbitrary switching signal α. Specifically, we obtain T a = 1. Besides, we can estimate a bound for the evolution of the state norm using (16).
Computing the eigenvalues of P, it is obtained ξ (kh) ≤ 1.0573 · (0.8944) k−k 0 ξ (k 0 h) . Note that if we set µ > 1, then we cannot guarantee arbitrary switching independently of the value of β. Besides, if (8)- (9) are only valid for µ > 1 and β → 1, then no switching is allowed, i.e., the MRRNN operates at one of the modes all the time, so (7) is transformed into (6). In Figure 2, we observe the evolution of the neurons, and how they are exponentially stabilized according the results of Theorem 1.
Simulation 2 (Synchronization Analysis): Consider now the switched MRRNN described in Example 1 and a second response switched MRRNN as shown in (19). If the feedback matrices K (α(k)) are known, then Theorem 2 can be used to study the synchronization. However, in this example we assume that K (α(k)) should be found. So, we can apply Corollary 1 to compute the feedback control law that synchronize the response system with the drive system. We set now µ = 1 and β = 0.4. Then, Corollary 1 provides the matrices     As in the previous example, the feasibility of the LMIs (25)-(26) for µ = 1 implies that the exponential synchronization is achieved for any arbitrary switching signal α.
Simulating the MRRNNs, we can observe this exponential synchronization. In Figures 3-4, the states of the neurons for the drive and the response networks are depicted. Figure 5 shows the error between the drive and the response network. Finally, we represent in Figure 6 the control signal applied to the response network in order to synchronize it with the drive network.

VII. CONCLUSION
In this work, we provide a general framework to describe switched MRRNNs, i.e., RNNs with different periods of activation of the neurons and whose weights can vary among different modes. This framework allows us to study the stability of a switched MRRNN and the synchronization between two of them. Therefore, it can be applied to different RNNs with multiple periods such as clockwork RNNs, hierachical multiscale RNNs or dLSTM.
In future works, we would like to extend the results to more complex cases, especially focusing on the communication between several networks, where problems such as time delays or packet dropouts usually appear. JULIÁN SALT received the M.Sc. degree in industrial engineering and the Ph.D. degree in control engineering from Valencia Polytechnic University, in 1986 and 1992, respectively. Since 1987, he has been a Full Professor of automatic control in ETSI design with Valencia Polytechnic University (UPV), teaching a wide range of subjects in the area from continuous and discrete simulation to automation and programmable logic controllers applications. He was a Visiting Scholar in multirate control of hard disk drives with UC Berkeley, USA. He was the Head of the Department of Systems Engineering and Control, UPV. He has been the director of eight Ph.D. dissertations. He has taken part in research projects funded by local industries, Spanish Government, and the European Science Foundation. He has coauthored more than 90 technical articles in journals and technical meetings. His research interests include non-conventionally sampled control systems and networked-based control systems. His research activity is currently involved in networked-based control energy saving.
SEBASTIÁN DORMIDO (Member, IEEE) received the B.S. degree in physics from Complutense University, Madrid, Spain, in 1968, the Ph.D. degree in sciences from Basque Country University, Bilbao, Spain, in 1971, and the joint Ph.D. degree (Hons.) from the Universidad de Huelva and Universidad de Almeria. In 1981, he was appointed as a Professor of control engineering with UNED, Madrid. From 2001 to 2006, he was the President of the Spanish Association of Automatic Control, CEA-IFAC. He has authored or coauthored more than 250 technical papers in international journals and conferences and has supervised more than 35 Ph.D. thesis. His research interests include computer control of industrial processes, model-based predictive control, hybrid control, and web-based labs for distance education. He received the National Automatic Control Prize from the IFAC Spanish Automatic Control Committee. VOLUME 9, 2021