A heterogeneous ensemble of trees

Decision Tree is a simple but popular machine learning algorithm. Although a single decision tree is not as accurate as other state-of-the-art classifiers, the performance can be significantly improved by combining the predictions of several decision trees i.e. by creating an ensemble of trees. In this paper, we study decision trees and their ensembles viz. Bagged Decision Trees, Random Forest, Extremely Randomized Trees, Rotation Forest, Gradient Boosted Trees and AdaBoosted Trees, and assess their performance on several UCI datasets. In addition, we propose a new ensemble method, Heterogeneous Ensemble of trees, and compare its performance with existing tree based classifiers. The heterogeneous ensemble is built with three different ensemble of trees (Random Forest, Rotation Forest, and Extremely Randomized Trees) with equal proportions to boost the diversity of the trees in the ensemble. A weightage scheme based on out-of-bag error is employed to combine the prediction of various trees for the final output prediction. Based on the experiments performed on several UCI datasets, the Heterogeneous Ensemble of trees obtains the best rank compared with other tree based classifiers.