A Study of Text Difficulty Assessment Based on Decision Trees

With the increase of information technology, stu-dents can receive more and more reading materials for learning, but how to choose from the huge amount of reading materials and find the right one for them efficiently and accurately has become a difficult problem for students. Therefore, quantitative assessment of text difficulty has become an important topic in the field of language research and education. In this paper, we first analyze the difficulty of English text, classify the text difficulty level into six difficulty levels: elementary school, junior high school, high school, university and subject specialization, and initially determine 27 difficulty quantification indexes in four aspects: vocabulary difficulty, sentence difficulty, difficulty of the whole corpus and speech chunks. The pre-processed text data were then quantified using the Range text quantization tool to generate the training dataset. The classification accuracy of the 27 difficulty indicators was then tested separately, and the different indicators were combined, and the classification accuracy of all the combined indicators was obtained using a genetic algorithm, and the ten combinations of indicators with the highest classification accuracy were finally determined. A decision tree based English text difficulty assessment model was built using the RapidMiner model building tool with the training dataset, and the model was analyzed by the validation results with good classification effects. Next, the model is extended to study the Chinese text difficulty assessment model. According to the characteristics of Chinese, we identified 21 indicators to assess the difficulty of Chinese text for comprehensive consideration, such as: the difficulty of Chinese characters, the difficulty of words, the difficulty of sentences, and the readability of the text as a whole. By collecting Chinese text corpus, a decision tree based Chinese text difficulty assessment model was established using the same method. Through model validation, it is shown that the model has good classification effect.