On-Edge High-Throughput Collaborative Inference for Real-Time Video Analytics

Performing video analytics tasks based on deep neural networks (DNNs) on resource-constrained mobile devices is extremely challenging because of the huge volume of video data and the computationally intensive nature of DNN. One promising solution is to offload tasks to the edge servers for execution. However, due to explosive growth in the number of end devices, more and more mobile devices are connected to the edge servers. This makes it difficult for the edge server to meet the specific service level objective (SLO) of on-edge video analytics when facing concurrent computing requests, especially in the real-time scene. To address this issue, this article presents EHCI, an on-edge high-throughput collaborative inference framework for real-time video analytics. On the mobile device, EHCI crops the key regions from the current video frame based on the local detection cache and offloads these regions to the edge server, which can significantly reduce bandwidth consumption and computation costs. Besides, considering concurrent DNN inference requests from multiple mobile devices, EHCI uses a key region patching method to achieve high-throughput DNN inference on the edge server, along with a scheduling algorithm to meet the SLO for each mobile device. It has been validated with testing that the EHCI outperforms the state-of-the-art technology by 159% in achieved throughput, reduces the average end-to-end delay by 36%, and the application accuracy sacrifice is within a reasonable range.