The problem difficulty impact on machine solving of math word problems

Over the last decade, the development of algorithms to solve mathematical problems automatically has garnered growing interest. Several deep learning models have achieved impressive performance on standard benchmark datasets. However, the newly released large dataset ape200k and the commonly used math23k dataset, among others, do not have difficulty ratings. As a result, the generalization ability of each model trained on these benchmark datasets remains questionable. This paper aims to determine the factors that contribute to question difficulty using data mining techniques. Further, it proposes a novel approach of utilizing hierarchical analysis to assign difficulty values to each question and achieve a scientific classification of the questions’ difficulty levels. The study also verifies the generalization ability of ten models, including the Goal-Driven Tree-structured model. The experimental results demonstrate that datasets of varying difficulties significantly impact the accuracy of the model’s automatic answers. Further, they reveal that essential factors, such as question length, answer length, implied meaning, and complexity, impact question difficulty.