Multi-Knowledge Distillation for Constructing Math Word Problem Encoder

Technology of solving math word problems is predicted to play an increasingly important role in smart education today. It will be used to help students master complex problem solving faster, promote personalized tutoring, and support the advancement of smart education. The encoder is a very important part of the math word problem solving algorithm. Particularly the BERT encoder based solving algorithms have a good accuracy. However, in educational environments, computational resources are usually limited, and pre-trained models like BERT are difficult to be applied in resource limited scenarios due to the large number of parameters and the huge amount of computation. Therefore, in order to promote the popularization of intelligent education, it is necessary to study small encoder applicable to math problem solving. Knowledge distillation is an effective method of model compression, so this paper applies it to the field of math word problems by combining the knowledge of the layers of the teacher model, as well as the math knowledge of the predictive label through different decoders and the true label to distill a lightweight encoder suitable to be used on small devices. Results of experiment show that with 90% reduction in the number of parameters, the answer accuracy only 11.9% lower than the original BERT in the math word problem solving task and significantly outperforms the existing mainstream small models. This provides strong support for the practical application of algorithms for solving math word problems.