CrossNet: A Low-Latency MLaaS Framework for Privacy-Preserving Neural Network Inference on Resource-Limited Devices

With the development of cryptographic tools such as Fully Homomorphic Encryption (FHE) and secure Multiparty Computation (MPC), privacy-preserving Machine Learning as a Service (MLaaS) has gained attractiveness for its security when it comes to utilizing cross-domain data. However, cryptographic tools are characterized by huge overhead, which results in the MLaaS quality being unbearably degraded, especially for latency-sensitive MLaaS applications. In this paper, we focus on the problem of low-latency inference associated with MLaaS and propose CrossNet, a Privacy-preserving Neural Network Inference (PPNI) framework based on FHE, for applications with limited client-side computational and communication resources. CrossNet performs model transformations on neural networks so that they can be evaluated in an FHE-friendly manner. Model transformation introduces limited interactions between client and server, thus restricting inference latency. In addition, CrossNet includes a series of layer constructions where elaborate encoding forms and computational orders are designed to further reduce the overhead of transformed layers. CrossNet outperforms the existing FHEbased frameworks by 4x efficiency and reduces nearly 30% inference latency on ResNet-50 in a resource-limited setting.