An Approach of Analyzing Classroom Student Engagement in Multimodal Environment by Using Deep Learning

Student engagement refers to the level of involve-ment, interest, attention, and active participation that students demonstrate during their learning experiences. As the engage-ment rate increases the learning rate, teachers always try to mea-sure the engagement rate in the classroom in various ways. This paper introduces a multimodal approach for detecting student engagement, a vital aspect of effective learning. Leveraging Convolutional Neural Networks (CNNs), video and audio data are integrated to capture comprehensive insights into engagement levels. The objective is to classify student engagement into three categories: highly engaged, engaged, and not engaged. Separate unimodal models are employed for speech and video data., achieving notable accuracies of 92.87 % and 94.77 %, respectively, in recognizing engagement levels. The combined model, achieved through feature-level fusion, yields a competitive accuracy of 69.36%, benefiting from the strengths of both modalities. Al-though the combined model exhibits slightly lower test accuracy compared to the individual speech and video models, its enhanced interpretability and comprehensive insights make it a promising avenue for future research.