Multimodal Analysis of Learning-Centered Emotions and Cognitive Processes in Open-Ended Learning Environments

Self-Regulated Learning (SRL) is a multifaceted process encompassing cognitive, affective, metacognitive, and motivational components. Understanding the interplay of these components, particularly emotions, is crucial for fostering effective learning behaviors. This study addresses the challenge of recognizing Learning-Centered Emotions (LCEs) in educational settings by reannotating the DAiSEE dataset (a dataset for learning-centric emotion with Indian students) to provide frame-by-frame emotion labels suitable for static image analysis. Using this reannotated dataset, we trained and validated several state-of-the-art deep learning models and fine-tuned the Inception ResNet V2 model, demonstrating superior performance. The fine-tuned model was applied to the MEttLE (Modelling-based estimation learning environment) open-ended learning environment to analyze the interplay between detected academic emotions and cognitive processes among six engineering students. The results revealed significant differences in emotional distributions among high, medium, and low performers. High performers exhibited higher engagement levels, while low performers showed more frequent transitions into boredom. A Chi-square test of independence indicated a strong association between students’ performance levels and their emotional responses, with significant differences across cognitive processes based on their performance (high, medium, low) in the given task in the environment. Additionally, our findings emphasize the importance of personalized interventions and the potential for intelligent tutoring systems to adapt in real-time to the emotional states of learners.