Ensemble Learning Based on the Output Sensitivity of Multilayer Perceptrons

Ensemble learning to construct learners in regression and classification has practically and theoretically been proved to be able to improve the generalization capability of the learners. Nowadays, most neural network ensembles are obtained by manipulating training data and networks' architecture etc, such as Bagging, Boosting, and other methods like evolutionary techniques. In this paper, a new method to construct neural network ensembles is presented, which aims at selecting, by means of output sensitivity of an individual network, the most diverse members from a pool of trained networks. Conceptually, the sensitivity reflects a network's output behavior at a given data point, for example, the trend of the network's output nearby. So the sensitivity can be helpful to explicitly measure the output diversity among individuals in the pool. In our research, Multilayer Perceptrons (MLPs) are focused on, and the sensitivity is adopted as the partial derivative of an MLP's output to its input at data point. Based on the sensitivity, we developed four different measures for the selection of the most diverse individuals from a given pool of trained MLPs. Some experiments on the UCI benchmark data have been conducted, and the comparisons of our results with those from Bagging and Boosting show that our method has some advantages over the existing ensemble methods in ensemble size and generalization performance.