Context-Aware Layer Scheduling for Seamless Neural Network Inference in Cloud-Edge Systems

With deep neural networks (DNNs) gaining popularity for tasks like object detection and image segmentation in domains like autonomous driving and smart agriculture, DNN inference on edge devices is increasingly necessary. While inference on edge devices can provide real time capabilities, these devices might be time and energy constraint. Offloading execution to the cloud can provide a possibility to save energy and execution time on the edge device, when a stable network connection is present. Contrary to that, failing to communicate with cloud servers during operation is detrimental to offloaded inference.In this work, we introduce a scheduling algorithm capable of leveraging the strength of cloud offloading in reducing execution time and power consumption on the edge device, while simultaneously being able to recover in the event of a network failure and continue inference seamlessly. Only raising the worst case execution time by the length of one branching interval, our method provides the ability to greatly reduce energy consumption and execution time. For FCN-ResNet50 inference on an edge accelerator, we were able to lower the overall execution time from 142.25 ms to 12.23 ms and the energy consumption on the edge device from 105.02 mJ to 5.46 mJ, while recovering from connectivity failures twice with continued inference. Comparable improvements can be achieved for other use cases using GoogleNet or SqueezeNet.