Leveraging Semantic Facets for Automatic Assessment of Short Free Text Answers

Free text answers to short questions can reflect students' mastery of concepts and their relationships relevant to learning objectives. However, automating the assessment of free text answers has been challenging due to the complexity of natural language. Existing studies often predict the scores of free text answers in a “black box” manner without analyzing their semantic components, which at least partially limit the prediction performance. In this article, we focus on fine-grained semantic facets in free text answers that correspond to knowledge to be mastered. Using a dataset with semantic facet annotation, we first show the correspondence of semantic facet matching states and answer quality, as well as the importance of semantic facets in automatic assessment of answer quality. We then extend the work to a dataset without semantic facet annotation and demonstrate the effectiveness of proposed automated methods in assessing answer quality, including semantic facet extraction, matching state prediction based on a neural framework, and feature engineering with semantic facets. The contribution of this research is twofold: 1) the proposed methods improve state-of-the-art performance of automatic assessment of free text answers and 2) it delves into fine-grained semantic components of free text answers, making it possible to explain the scores and generate detailed feedback.