PPKD: Privacy-preserving Knowledge Distillation for Large Model

With the development of deep learning technology and the application of large-scale models, high training costs and a large number of model parameters have become bottlenecks that limit technological development. To address these issues, existing model distillation techniques can transfer knowledge from large models to small models, reducing the model size and computational resource consumption while maintaining high performance. However, existing research has overlooked the privacy protection of input data and large models during the distillation process. In practical scenarios, the entities who train student models and own teacher models may be different institutions or countries, and how to protect the privacy of training data and teacher models is a challenge facing cross-institutional distillation learning. To solve this problem, this paper proposes a privacy-preserving model distillation method combining random masking and threshold encryption systems. We introduce noise based on random masking into training data and encrypt the output of the teacher model, ensuring the privacy of the input data and teacher model while ensuring the performance of the distilled student model. We rigorously prove the security and correctness of the scheme in theory and validate its effectiveness through experiments. The experimental results show that the student model trained after data protection has a similar classification performance to the student model in the original distillation learning.