Model selection and local optimality in learning dynamical systems using recurrent neural networks

We consider learning a dynamical system (DS) by a continuous-time recurrent neural network (RNN). An affine RNN (A-RNN), whose hidden units are linearly related to visible ones, is defined so that it always produces a DS. Learning a DS by an A-RNN is performed as a three-layer perceptron. The paper investigates model selection and the local optima problem in learning. The experiments showed that model selection can be exactly done by monitoring generalization performance and in the learning there exist much more local optima than expected.