Deep Stacked Random Forest

Current deep forest are mostly built upon multi-grained cascade forest, i.e. a novel decision-tree ensemble, with a cascade structure that enables representation learning by forest. In this paper, we propose the Deep Stacked Random Forest(DSRF) based on Deep Forest. We have made the following three enhancements: (1) The enhancement of data learning strengthens the cascade part of the deep forest model. DSRF applies a Random Feature Extraction (RFE) method to obtain more diverse feature subsets from the original data, which benefits feature learning, thus laying a solid foundation for the performance improvement of the model. (2) Designing the Mean Stacked Forest(MSF), which adopts mean output, reduces the impact of part of the classifiers reaching local optima on the accuracy of the algorithm, reduces the convergence speed of the representation learning model. To a certain extent, MSF reduces memory consumption. (3) Adding the null space matrix on training data promotes the model’s ability to learn mapping rules from the input data, thereby improving the generalization and fault tolerance of the model. Experiment results show that DSRF achieves advanced performance on four low-dimensional datasets and two high-dimensional datasets. Especially, the model achieves state-of-the-art performance in terms of convergence speed and learning efficiency on low-dimensional datasets.