Falcon: Accelerating Homomorphically Encrypted Convolutions for Efficient Private Mobile Network Inference

Efficient networks, e.g., MobileNetV2, EfficientNet, etc, achieves state-of-the-art (SOTA) accuracy with lightweight computation. However, existing homomorphic encryption (HE)-based two-party computation (2PC) frameworks are not op-timized for these networks and suffer from a high inference overhead. We observe the inefficiency mainly comes from the packing algorithm, which ignores the computation character-istics and the communication bottleneck of homomorphically encrypted depthwise convolutions. Therefore, in this paper, we propose Falcon, an effective dense packing algorithm for HE-based 2PC frameworks. Falcon features a zero-aware greedy packing algorithm and a communication-aware operator tiling strategy to improve the packing density for depth wise convo-lutions. Compared to SOTA HE-based 2PC frameworks, e.g., CrypTFlow2, Iron and Cheetah, Falcon achieves more than 15.6 x, 5.1 x and 1.8 x latency reduction, respectively, at operator level. Meanwhile, at network level, Falcon allows for 1.4 % and 4.2% accuracy improvement over Cheetah on CIFAR-100 and Tiny Imagenet datasets with iso-communication, respecitvely.