Stable limit cycles in recurrent neural networks

The goal of this paper is to contribute to the understanding of the dynamics of recurrent neural networks. Specifically, we establish conditions for the existence of stable limit cycles, whose existence is equivalent to the echo state property. We provide sufficient conditions for the convergence to a trajectory that is uniquely determined by the driving input signal, independently of the initial states. Under these conditions, the hidden-to-hidden matrix may have norm larger than one. This result can help extending the memory of recurrent neural networks, since earlier work has shown that large matrix norms in the hidden layer imply longer memory duration. This would also increase the design options for recurrent neural networks.