FasterBERT with Double Loss Function Based on Batch Data Preprocessing

Even though pre-trained language models like BERT and XLNet have produced significant consequences on a variety of tasks of natural language processing, they are difficult to deploy in practical applications due to their large number of parameters, and huge calculation and memory costs. In order to improve the training speed and inference speed of the pre-trained model, the FasterBERT model is proposed in this paper. This model combines batch data preprocessing method with knowledge distillation. It increases the model to increase the model speed and maintains the original accuracy. Furthermore, double loss function is incorporated during the knowledge distillation process in this paper, so that the model can better learn the knowledge of the teacher model and the knowledge of the real labels. The accuracy of the model is improved. The effectiveness of the FasterBERT model is demonstrated on 12 datasets in this paper. Training speed of the FasterBERT is 1.1-2.2 times higher than FastBERT and 2-43 times higher than BERT. In addition, the accuracy is up to 0.6% higher than that of FastBERT, and the accuracy is better than that of the BERT model on three datasets. The experiments show that the FasterBERT model proposed in this paper performs well on six Chinese datasets and six English datasets. The speed is greatly improved while the accuracy of the task also has a significant effect.