Understanding the Runtime Overheads of Deep Learning Inference on Edge Devices

With the growing ubiquity of the Internet of Things, in-the-edge inference of deep neural network models has been a major driver for promoting the widespread use of intelligent applications. As model inference characteristics are crucial for optimizing and deploying deep neural networks on hardware platforms, many studies focus on analyzing the performance of neural networks such as latency, accuracy, throughput, and energy consumption. However, few existing works have ever discussed the runtime overheads hidden in neural network inference, despite the overheads are non-negligible for edge applications. The lack of in-depth analysis of the overheads hinders the understanding of how hardware designs and model structures impact on-device inference performance. In this paper, we characterize the runtime overheads of deep learning inference on representative edge devices by leveraging state-of-the-art neural network models, performing a systematical analysis from the perspective of end-to-end performance, hardware platforms, memory bandwidth, and neural network model structures. Based on experimental results, the crucial insights are offered to facilitate the design and configure of resource-efficient networks and pick appropriate models on the specific platform, which provides a comprehensive view of runtime overheads of in-the- edge neural network inference for architects and developers.