SPOT: Structure Patching and Overlap Tweaking for Effective Pipelining in Privacy-Preserving MLaaS with Tiny Clients

Machine Learning as a Service (MLaaS) has paved the way for numerous applications for resource-limited clients, such as IoT/mobile users. However, it raises a great challenge for privacy, including both the data privacy of clients and model privacy of the server. While there have been extensive studies on privacy-preserving MLaaS, a direct adoption of current frameworks leads to intractable efficiency bottleneck for MLaaS with resource constrained clients. In this paper, we focus on MLaaS with resource constrained clients and propose a novel privacy-preserving framework called SPOT to address a unique challenge, the memory constraint of such clients, such as IoT /mobile devices, which results in significant computation stalls at the server in privacy-preserving MLaaS. We develop 1) a novel structure patching scheme to enable independent computations for sequential inputs at the server to eliminate the computation stall, and 2) a patch overlap tweaking scheme to minimize overlapped data between adjacent patches and thus enable more efficient computation with flexible cryptographic parameters. SPOT demonstrates significant improvement on computation efficiency for MLaaS with IoT /mobile clients. Compared with the state-of-the-art framework for privacy-preserving MLaaS, SPOT achieves up to 2 × memory utilization boost and a speedup up to 3 × on computation time for modern neural networks such as ResNet and VGG.