Dynamic Hierarchical Neural Network Offloading in IoT Edge Networks

In recent developments in machine learning, a trend has emerged where larger models achieve better performance. At the same time, deploying these models in real-life scenarios is difficult due to the parallel trend of pushing them on end-users or IoT devices with strong resource limitations. In this work, we develop a novel technique for executing parts of a single model successively through multiple devices (IoT, edge, cloud) while respecting each device’s resource limitations. For that, we introduce a new offloading mechanism where, during computation, a decision can be made to offload work, together with the ability to exit early in the computation with intermediate results. The decision itself is tuned through Deep Q-Learning.