Orthogonal Least Squares Based Incremental Echo State Networks for Nonlinear Time Series Data Analysis

This paper develops an incremental randomized learning method for an extended Echo State Network (φ-ESN), which has a reservoir with random static projection, to better cope with non-linear time series data modelling problems. Although the typical ESN can effectively improve the prediction performance of the network by extending a random static nonlinear hidden layer, since the input weights and biases of the hidden neurons in the extended static layer are randomly assigned, some neurons have little effect on reducing the model error, resulting in high model complexity, poor generalization and large performance fluctuation. A constructive incremental randomized learning method termed OLS-φ-ESN is proposed for generating the nodes of the extended static nonlinear hidden layer. Two-step training paradigm is adopted, namely, randomly assigning the input weights and biases of the hidden neurons in the extended static layer according to a supervisory mechanism and solving output weights by least squares algorithm. Based on Orthogonal Least Squares (OLS) search algorithm, the proposed supervisory mechanism is designed where an adaptive threshold is also set to better control the compactness of the generated learner model. Simulation results concerning both nonlinear time series prediction and system identification tasks indicate some advantages of our proposed OLS-φ-ESN in terms of more compact model and sound generalization.