A unifying view of stochastic approximation, Kalman filter and backpropagation

In this paper the relationships between the stochastic approximation, the Kalman filter and the backpropagation algorithms are investigated. We show that when the neural network architecture at hand can be formalized such that the approximation of the optimum for a nonlinear objective function is the problem for which we seek a solution, then both stochastic approximation techniques and appropriate Kalman filters can be employed in order to reach the goal but the latter can also handle various structural characteristics of the stochastic processes involved and suggest a more efficient two-step estimator.