SecureGPT: A Framework for Multi-Party Privacy-Preserving Transformer Inference in GPT

Generative Pretrained Transformer (GPT) is an advanced natural language processing (NLP) model and is excellent at understanding and generating human language. As GPT is increasingly utilized, more and more cloud inference services for pre-trained generative models are being offered. However, when users upload their data to cloud servers to experience cloud inference services, ensuring the privacy and security of their data becomes a challenge. Thus, in this work, we present SecureGPT, a framework for multi-party privacy-preserving transformer inference in GPT and design a series of building blocks which include M2A (conversion of multiplicative share to additive share), truncation, division, softmax and GELU protocols for our framework. Specifically, we follow the work of SecureNLP and further explore the M2A protocol for non-linear functions such as GELU and softmax. We also design multi-party private protocols for GPT’s transformer sub-layers. Finally we prove the security of our framework in the semi-honest adversary model with all-but-one corruptions. we evaluate the runtime of our framework under different parties settings and our implementation leads to up to $100\times $ improvement compared to state-of-the-art works.