Loading web-font TeX/Main/Regular
Towards Efficient Inference: Adaptively Cooperate in Heterogeneous IoT Edge Cluster | IEEE Conference Publication | IEEE Xplore

Towards Efficient Inference: Adaptively Cooperate in Heterogeneous IoT Edge Cluster


Abstract:

New applications such as smart homes, autonomous vehicles are leading an increasing research topic of convolutional neural network (CNN) based inference on IoT edge devic...Show More

Abstract:

New applications such as smart homes, autonomous vehicles are leading an increasing research topic of convolutional neural network (CNN) based inference on IoT edge devices. Unfortunately, this scenario meets a huge roadblock caused by the limited computing resources owned by these devices. One popular solution is to execute inference on an edge cluster with parallelization schemes instead of on a single device. However, the heterogeneous edge devices and varied neural layers bring challenges to this process. In this paper, we propose a pipelined cooperation scheme (PICO) to efficiently execute CNN inference for edge devices. Our goal is to maximize throughput by reducing redundant computing meanwhile to keep the inference latency under a certain value. PICO divides the neural layers and edge devices into several stages. The input data is fed into the first stage and the inference result is produced at the last stage. These stages compose an inference pipeline. The execution time of each stage is optimized to approach the maximum throughput as close as possible. We also implement an adaptive framework to choose the best inference scheme under different workloads. In our experiment with 8 RaspberryPi devices, the average inference latency can be reduced by 1.7\sim 6.5\times under different workloads, and the throughput can be improved by 1.8\sim 6.2\times under various network settings.
Date of Conference: 07-10 July 2021
Date Added to IEEE Xplore: 04 October 2021
ISBN Information:

ISSN Information:

Conference Location: DC, USA

Funding Agency:


References

References is not available for this document.