Auto-Scoring of Math Self-Explanations by Combining Visual and Language Analysis

In the field of mathematics education, self-explanation is recognized as a critical facilitator for learners to articulate their understanding of complex mathematical concepts and problem-solving techniques. With the emergence of digital learning platforms, the potential to utilize such self-explanations for automated evaluation has expanded, yet significant challenges remain. This study introduces a method that integrates vision and language models to enhance the accuracy of automated evaluations of self-explanations in mathematics quizzes. By leveraging the CLIP encoder, we utilize features from both handwritten images and textual self-explanations, aiming to incorporate the characteristics of handwritten solutions that have been overlooked by text-only evaluations. Models were developed to include self-explanations alone (baseline) and those that integrate image features, using both the original and a fine-tuned CLIP encoder adapted to our dataset of self-explanations and handwritten images. Experimental results demonstrated that the model utilizing the fine-tuned CLIP significantly outperformed the baseline, showing a notable reduction in MAE. Conversely, the model employing the original CLIP encoder exhibited decreased performance compared to the baseline, revealing the complex interplay between integrating self-explanations and image features. These findings suggest that the benefits of embedding image features depend on the quality and appropriateness of the visual data incorporated.