Accelerating Decision-Tree-Based Inference Through Adaptive Parallelization

Gradient-boosted trees and random forests are among the most popular machine learning algorithms. They are applied to a broad spectrum of applications as diverse as search engine ranking, credit card fraud detection, customer relationship management, fault diagnosis in machinery, and geological studies, to name a few. Inference of these models is supported by many widely-used libraries and frameworks including Scikit-Learn, XGBoost, and ONNX Runtime. Many of the inference algorithms integrated in these libraries are optimized for performing fast predictions for large input batches, often targeted at ensemble models containing relatively shallow decision trees. This does not necessarily match well with the requirements of new emerging applications that depend on real-time predictions for individual samples or small input batches. Also, cloud-based inference services put more emphasis on small memory footprints. This paper aims to fill this gap by proposing a novel inference scheme for decision-tree ensembles that efficiently handles a wider range of application requirements and model characteristics. Performance is maximized for an extensive number of parameter combinations through a new concept in which a predict function is selected dynamically at runtime. This is done in a way that the processing resources, including the use of SIMD vector processing units and multithreading, are optimally exploited. Experiments have shown substantial performance improvements over state-of-the-art algorithms for common models.

I. Introduction

Decision trees in machine learning have a long history. Automatic Interaction Detection (AID) [1] and THeta Automatic Interaction Detection (THAID) [2] are often considered to be the first published decision tree algorithms for regression and classification, respectively, which were combined and extended into the Classification And Regression Trees (CART) algorithm [3]. Later, ensemble learning methods such as bagging [4], gradient boosting [5]–[7] and random forests [8], [9] were developed, which combine multiple decision trees to substan-tially improve prediction accuracy over individual decision trees. Despite this long history, decision trees and ensemble methods such as random forests and gradient boosting are still among the most frequently used machine learning algorithms today, as shown by a recent Kaggle survey [10]. This is because of the inherent advantages coming from the relatively simple concept, support for numerical and categorical features and interpretability of predictions.