Bayesian Decision-Theoretic Prediction with Ensemble of Meta-Trees for Classification Problems

Decision tree algorithms are one of the most popular methods in machine learning. However, most decision tree algorithms do not assume a stochastic model behind data. On the other hand, a meta-tree was recently proposed as a stochastic model with a tree structure. The prediction under the assumption of the meta-tree is decided using Bayesian decision theory. Although the optimal prediction can be calculated with an assumption of a known meta-tree, an approximation is necessary to obtain a prediction under the problem setting of an unknown meta-tree because of the marginalization of all possible meta-trees. In this paper, we propose an approximation method, where a subset of meta-trees is sequentially constructed, and the prediction is made by weighting the meta-trees. For the approximation, we clarify the approaches of constructing the subset and making predictions with weighted meta-trees. We also examine the effectiveness of the approaches in an experiment using synthetic data. In addition, we conduct an experiment on benchmark data to confirm the performance of the proposal.