Universality and Approximation Bounds for Echo State Networks With Random Weights

We study the uniform approximation of echo state networks (ESNs) with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that ESNs with ReLU activation are universal. In this article, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the ESN can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU ESNs for sufficiently regular operators.