Complementary prioritized ensemble selection

We present a complementary ensemble selection method that utilizes a novel priority queue-based diversity measure. The method considers voting weaknesses of the current ensemble in covering the training set, and finds a classifier that can remove the highest priority weaknesses. Individual classifiers are generated using different machine learning algorithms and different parameter settings. A key feature of our method is in selecting complementary classifiers, i.e., repeatedly adding individual classifiers that best complement incorrect voting patterns of the current ensemble in priority order. We refer to this approach as “complementarity”. To evaluate this approach, we have performed months of experiments, making comparisons between our complementary prioritized ensemble selection method and an “ensemble of ensembles” approach. The comparisons are based on the forward stepwise selection method from earlier work on 5 datasets, each of which consists of comparisons from 9 folds of validated data (from 2 fold to 10 fold cross validated data). Over 1600 different classifier types were considered, yielding a huge space of alternative ensembles. The experimental results showed that a small and compact complementary ensemble yielded performance as good as and sometimes better than a huge ensemble selected by a state-of-the-art Ensemble of Ensembles method.