Optimizing Complex Ensembles: Automatic Bias-Variance Tuning with SVM-Based Learners

Ensembling is an extremely powerful and popular method for increasing the strength of a machine learning predictor by combining it with other predictors in order to obtain more accurate results. However, it is difficult for ensembles to strike a balance between bias and variance, and between performance and diversity among candidate models during ensemble constructions. To help alleviate this problem, a novel ensemble construction method and a dynamic weighting strategy is proposed. During the ensemble construction, candidate models are selected by a mixture of their individual performances and their agreement with the preexisting ensemble model. When obtaining final predictions, base learners in the ensemble are assigned weights that are dependent on their individual performance and how similar the test data is to the data that the model has performed uniquely well on. Experiments on real-world datasets show that this method performs extremely well across a wide variety of domains.