On Bayesian Learning and Stochastic Approximation

A general mathematical formulation for learning in an unknown stationary environment is established from the viewpoint of stochastic approximation. The existing learning techniques, based on the Bayesian type of inference, are first shown to fall into the general framework of stochastic approximation algorithms. As a consequence of the modeling, some convergence properties, optimal characteristics, and possible improvements of the learning schemes are then derived with less effort and fewer restrictions. The results of this work may provide an alternative approach to the study of learning theory and suggest a different mathematical basis for the analysis and synthesis of learning systems in pattern recognition, automatic control, and statistical communications.