A Distributed Learning Method for Deep Echo State Network Based on Improved SVD and ADMM

In recent years, echo state network (ESN) has emerged as an effective way in processing the classification and regression of time series. As a combination of deep learning and ESN, deep echo state network (DESN) naturally improves the performance of shallow ESN. However, two key challenges have to be addressed to achieve the training of DESN in edge environments. On the one hand, a high-dimensional reservoir state matrix is generated when training DESN with a large amount of data, which causes to the high computation cost. On the other hand, it probably produces the delay, privacy concerns when transmitting data from distributed edge nodes to a centralized location. To solve the above challenges, we propose a distributed learning method for DESN based on improved singular value decomposition (SVD) and alternating direction method of multipliers (ADMM). First, a decomposition algorithm for reservoir state matrix based on improved SVD is proposed to train DESN only using the main features of reservoir state matrix, thus reduces the computation cost. Second, a distributed training algorithm for DESN based on improved ADMM is proposed by using dispersed data to train DESN fast, which produces the less communication cost. Finally, the experimental results in theoretical and real-world datasets show that the proposed method can effectively reduce the computation and communication costs in the learning process of DESN while suffering an acceptable performance loss, compared with other distributed learning methods.