Recurrent Expectation Maximization Neural Modeling

A probabilistic approach to training recurrent neural networks is developed for maximum likelihood estimation of network weights, model uncertainty, and noise in the data. We elaborate on an Expectation Maximization algorithm where by a forward filtering backward smoothing framework is utilized for estimation of network weights in the Expectation step, and in the Maximization step, the model uncertainty and measurement noise estimates are computed. Experimental investigations on real world data sets show that the developed algorithm outperforms the standard real time recurrent learning and extended Kalman Filtering algorithms for recurrent networks, as well as other contemporary nonlinear models, on time series modeling tasks.