A Theory of Bayesian Learning Systems

Efforts are made to simplify the implementation and to improve the flexibility of Bayesian learning systems. Using a truncated series expansion to represent a pattern class, a simplified structure is shown with nearly optimal performance. A criterion of determining the learning sample size is given so that after taking a sufficient number of learning observations, the system may elect to learn by itself without relying on the external supervision. A time-varying random parameter is approximated by the polynomial with random coefficients. The Bayes estimates of the coefficients are obtained sequentially from the useful information in the learning observations. The condition for convergence of the unsupervised learning is established and shown to be closely related to the selection of the characteristic features. The system retains the same structure in both supervised and unsupervised learning processes with either the stationary or the time-varying random parameter.