Application of Auxiliary Models in Neural Network Knowledge Distillation and Gap Reduction Strategies

With the development of deep learning and advances in the hardware industry, deep neural network with large amounts of parameters has ushered in better and more accurate prediction results, but also increases GPU resource costs. In this way, the deployment of models in terminals become a new problem, deduction time also costs a lot. To address these issues, we propose a model-assisted approach on feature-based knowledge distillation by introducing an auxiliary model together with the teacher model to guide the learning of the student model. We use auxiliary model to reducing the gap caused by structural and scale differences between the teacher model and the student model. The result shows that the prediction accuracy improved significantly also maintain a good generation ability. The choice of student models still remains problem in the field of knowledge distillation still, and the differences in the structure of the teacher model student models are still the important factors affecting the results of knowledge distillation. Therefore, this paper focuses on the accuracy and generation ability of student model on the distillation results, based on which a model-assisted boosting method is proposed, in which the knowledge are migrated to the student model by the teacher model together with the auxiliary model.