EMMFormer: Secure Transformer Inference with Efficient Matrix Multiplication

Transformer has been widely applied across various domains due to its outstanding performance, driving the development of numerous application services. However, in many real-world scenarios, transformer-based services pose risks of data privacy leakage, especially when handling sensitive information. In this context, secure inference on fully homomorphic encryption (FHE) has rapidly evolved as an effective solution for protecting data privacy. However, extending FHE-based secure inference to transformers significantly increases inference latency, limiting their practicality. To address this issue, we propose EMMFormer, an efficient and secure transformer inference framework designed to enhance inference performance while ensuring data privacy. EMMFormer introduces a novel FHE-based efficient matrix multiplication method and optimizes nonlinear function computations in the transformer, significantly reducing computational overhead and improving the execution efficiency of secure transformer inference. Finally, we conduct extensive experiments on THUCNews and the Sogou News Topic Classification Dataset to verify and evaluate the performance of EMMFormer comprehensively.