Manto: A Practical and Secure Inference Service of Convolutional Neural Networks for IoT

As convolutional neural networks (CNNs) exhibit remarkable performance in various inference tasks, it is increasingly important to enable Internet of Things (IoT) devices to perform CNN-based applications. Many companies provide their carefully trained neural networks as inference services for resource-constrained clients (e.g., IoT devices). However, the use of CNN inference in many IoT applications raises privacy concerns. Cryptographic inference services provide a way to perform neural inference efficiently and, at the same time, preserve both the privacy of the client’s input data and the server’s proprietary model. Unfortunately, the existing solutions incur severe latency costs, stemming mostly from nonlinear activations such as ReLUs, which make them still unsuitable for deployment in real IoT devices. In this article, we propose Manto, a secure inference system of CNNs for IoT. Manto makes the following two specific efforts by combining the insights of machine learning and cryptography. First, we customize different quadratic activation functions to replace specific ReLU layers and further propose a sliding-window-based fine-tuning method to produce CNN models involving no or few ReLUs. These techniques allow us to speedup cryptographic inference and guarantee inference accuracy. Second, we develop a series of cryptographic protocols that support ReLU activations and its approximation variants (i.e., polynomial activations), which purely rely on the lightweight secret sharing techniques in the online execution and can well cope with the above-mentioned optimized CNN models in the ciphertext domain. Our experimental results show Manto obtains state-of-the-art performance, reducing online inference latency by
$66.2\%\sim 87.7\%$
over prior works on CIFAR-100 and TinyImageNet data sets.