Ensemble learning using observational learning theory

In this paper, we propose an ensemble learning algorithm, which we call the observational learning algorithm (OLA), motivated from the observational learning theory suggested by Bandura (1971). According to the theory, in a group of children who have different and insufficient knowledge for a task, each child can lean how to do the task by observing other children. In the OLA, a neural network ensemble is regarded as a group of children. Each network is trained with the virtual data that are generated from observing other networks as well as the bootstrapping data from the original data set. The virtual data function as both temporal hints having the auxiliary information about the target function and a regularization penalty for making networks smooth. From numerical experiments involving both regression and classification problems, the OLA is shown to give better generalization performance than simple committee and bagging approaches when insufficient and noisy data are given.