Dynamic Decision Tree Ensembles for Energy-Efficient Inference on IoT Edge Nodes

With the increasing popularity of Internet of Things (IoT) devices, there is a growing need for energy-efficient machine learning (ML) models that can run on constrained edge nodes. Decision tree ensembles, such as random forests (RFs) and gradient boosting (GBTs), are particularly suited for this task, given their relatively low complexity compared to other alternatives. However, their inference time and energy costs are still significant for edge hardware. Given that said costs grow linearly with the ensemble size, this article proposes the use of dynamic ensembles, that adjust the number of executed trees based both on a latency/energy target and on the complexity of the processed input, to tradeoff computational cost and accuracy. We focus on deploying these algorithms on multicore low-power IoT devices, designing a tool that automatically converts a Python ensemble into optimized C code, and exploring several optimizations that account for the available parallelism and memory hierarchy. We extensively benchmark both static and dynamic RFs and GBTs on three state-of-the-art IoT-relevant data sets, using an 8-core ultralow-power System-on-Chip (SoC), GAP8, as the target platform. Thanks to the proposed early stopping mechanisms, we achieve an energy reduction of up to 37.9% with respect to static GBTs (8.82 uJ versus 14.20 uJ per inference) and 41.7% with respect to static RFs (2.86 uJ versus 4.90 uJ per inference), without losing accuracy compared to the static model.