Ensemble classification based on Random linear base classifiers

We propose a simple ensemble classification algorithm, which employs a set of N randomly generated linear classifiers, followed by a selection process based on the performance of these classifiers on the whole set of training data. The top n performers are then linearly combined to form the final classifier. We analyze the VC dimension of the resulting hypothesis set from such a construction procedure, and show that it can be controlled by choosing the parameters N and n. The proposed algorithm enjoys low computational complexity, and for the MNIST dataset and several UCI datasets that we tested, the algorithm compares favorably in generalization error rate or running time to competing algorithms including Random Kitchen Sinks and AdaBoost.