DECC: Delay-Aware Edge-Cloud Collaboration for Accelerating DNN Inference

Deep neural network (DNN)-enabled edge intelligence has been widely adopted to support a variety of smart applications because of its ability to preserve privacy and conserve communication efficiency. The dilemma is that DNN models can be too large to be deployed on computationally constrained edge devices, and the volume of raw data can be too large to be efficiently transmitted to a centralized server. Thus, it is of utter importance that edge devices and cloud servers collaborate with each other to achieve fast and dependable model inference. Current collaborative solutions separate the DNN into two parts, which are placed and executed at the edge and in the cloud, respectively. However, these separated parts are executed consecutively, and all subsequent layers have to wait for the output of the previous layer even if they are not directly connected, causing significant inference latency. We propose a delay-aware edge-cloud collaboration (DECC) algorithm to reorganize the execution of DNN layers. By dividing DNN into several independent branches and selecting the optimal partition points, we apply a pipeline approach to parallelize the execution of these branches to minimize the inference delay. Extensive experiments show that the DECC outperforms existing methods by significantly reducing inference latency and improving throughput.