Memory Regressor Extended Echo State Networks for Nonlinear Dynamics Modeling

Echo state network (ESN) implements an alternative paradigm called reservoir computing to train recurrent neural networks (RNNs), where internal weights are randomly generated and kept fixed, and only readout weights need to be trained, which greatly reduces the training complexity of RNNs. ESN not only facilitates the practical implementation of RNNs but also shows superior performance over fully trained RNNs across a range of applications. However, the conventional ESN suffers from the drawbacks of stringent conditions for weight convergence and slow convergence speed. This paper proposes a memory regressor extended learning method to update the readout weights of ESNs. By constructing and incorporating a generalized prediction error based on regressor extension and filtering, the capacity of ESN to utilize historical data can be greatly improved. In the discrete-time domain, it is proven that exponential convergence of readout weights is achieved under a condition termed interval excitation that is strictly weaker than the classical condition of persistent excitation. Simulation results on modeling a 10th-order nonlinear autoregressive moving-average (NARMA) system have revealed that the proposed approach accelerates weight convergence speed almost ten times higher compared to the conventional ESN.