Latency- and Privacy-Aware Convolutional Neural Network Distributed Inference for Reliable Artificial Intelligence Systems

Reliable artificial intelligence (AI) systems not only propose a challenge on providing intelligent services with high quality for customers but also require customers’ privacy to be protected as much as possible during process of the services. Given the ultrahigh computing load brought by deep-learning-based intelligent services to edge devices and the ultralong distance between edge and cloud, low-latency requirement of intelligent services is hard to meet in single edge computing or cloud computing. Edge–cloud collaborative inference of deep neural networks (DNNs) is considered a feasible solution to the problem. However, former work has not reduced the inference latency to the greatest extent and has not considered privacy protection in distributed systems. To solve the problem, we first establish a novel queue mechanism. Then, the convolution layer split decisions are made based on deep reinforcement learning (DRL) to realize the parallel inference of convolutional neural networks (CNNs) for inference latency reduction. Next, for each CNN, the partition decision is made based on brute force algorithm to further reduce inference latency and protect customers’ privacy. Finally, simulation results show that our method performs better than the existing other methods.

I. Introduction

Nowadays, intelligent services based on deep learning (DL) are becoming more popular with the development of artificial intelligence (AI) and communication technology. As one of the main deep neural networks (DNNs), convolutional neural networks (CNNs) are widely used in vision applications such as object detection, object classification, and visual reality [1], [2], [3]. Considering the relatively less computing resources, little storage space and limited power of end devices and edge servers [4], it is difficult to deploy and infer a whole CNN model with large parameter quantity and computation amount completely with high efficiency on an edge device [5], [6]. One traditional solution to the problem is to offload the inference task of the CNN to the cloud server. With so strong computing power, the cloud server is able to infer a large CNN model with low latency and improve quality of intelligent services (QoSs).