Two constructive algorithms for improved time series processing with recurrent neural networks

Because of their universal approximation capabilities, recurrent neural networks are an attractive choice for building models of time series out of available data. Medium- and long-term dependencies are easier to learn when the recurrent network contains time-delayed connections. We propose two constructive algorithms which are able to choose the right locations and delays of such connections. To evaluate the capabilities of these algorithms, we use both natural data and synthetic data having built-in time delays. We then compare the two algorithms in order to define their domain of interest. The results we obtain on several benchmarks show that, by selectively adding a few time-delayed connections to recurrent networks, one is able to improve upon the results reported in the literature, while using significantly fewer parameters.