Automated Explanation Quality Assessment based on Similarity Calculations of Representative Explanations of Different Qualities

Peer instruction is instructional in guiding students to learn by answering questions, and explaining and discussing their answers with peers. Researchers recommended asking students to write down their answers and explanations before discussion to prevent social loafing. In addition, text-based explanations can be recorded and analyzed. The quality of students’ explanations varies, ranging from superficial and low-quality to detailed and in-depth high-quality explanations. In tradition, the qualities of students’ explanations were assessed by experts. Recently, machine learning classification models have been developed and applied to classify texts. However, the level of explanations of questions are question-dependent. Thus, each question needs its classification model. Therefore, a feature transformation was applied in this study so that the explanations of different questions could be combined and applied to train the same classification model. An automated explanation quality assessment mechanism was developed based on the similarity of representative explanations of different qualities. Students’ text-based explanations were collected and assessed by experts into four levels, ranging from 0 (worst) to 3 (best). The four-level classifications were merged into binary classifications of low (0 and 1) and high (2 and 3). Different classification models, including Support Vector Machine (SVM), Naive Bayes (NB), K Nearest Neighbor (KNN), Logistic Regression (LR), Random Forest (RF), and Bidirectional Encoder Representations from Transformers (BERT) were applied to train models and evaluate the accuracy of the models. In addition, three ensemble learning algorithms, including voting, stacking, and boosting, were applied to combine models chosen from SVM, NB, KNN, LR, and RF. The results showed that RF and RF+KNN+NB with stacking model showed the best accuracy (75.3%) among all four-level classification models whereas RF with boosting model showed the best accuracy (90%) among all binary classification models.