Multi-Target Regression Rules With Random Output Selections

In this article, we address the task of multi-target regression (MTR), where the goal is to predict multiple continuous variables. We approach MTR by learning global models that simultaneously predict all of the target variables, as opposed to learning a separate model for predicting each of the target variables. Specifically, we learn rule ensembles by generating many candidate rules and assigning them weights that are then optimized in order to select the best performing subset of rules. Candidate rules are generated by transforming ensembles of generalized decision trees, called predictive clustering trees (PCTs), into rules. We propose to extend an existing multi-target regression rule learning method named FIRE by learning tree ensembles that use random output selections (ROS). Such ensembles force individual PCTs to focus only on randomly selected subsets of target variables. The rules obtained from the tree ensemble also focus on various subsets of the target variables (FIRE-ROS). We use three different ensemble methods to generate candidate rules: bagging and random forests of PCTs, and ensembles of extremely randomized PCTs. An experimental evaluation on a range of benchmark datasets has been conducted, where FIRE-ROS is compared to three interpretable methods, namely predictive clustering rules, MTR trees and the original FIRE method, as well as state-of-the-art MTR methods, in particular ensembles of extremely randomized PCTs with ROS, random linear combinations and extremely randomized MTR trees with random projections of the target space. The results show that FIRE-ROS can improve the predictive performance of the FIRE method and that it performs on par with state-of-the-art (non-interpretable) MTR methods.