A Multimodal Approach for Real-Time Engagement Monitoring in E-Learning Using Machine Learning

This full research paper describes a multimodal approach to detect student engagement in an e-learning scenario. Remote learning has transformed access to education and helped reshape learning by offering students expanded access to resources, while also enabling educators to reach several new demographics. It has rightly been identified as one of the Grand Challenges for improving life on our planet. Unfortunately, gauging learner engagement has remained a significant challenge, limiting improvements and optimization of teaching methods. Machine Learning (ML), despite being an underutilized tool in the domain of e-learning, holds substantial potential for the real-time recognition of students' behavioral patterns, thereby helping provide personalized feedback to educators. This research paper aims to explore the potential of ML to compare multiple modalities such as facial action units (AUs), gaze vectors, and emotion indicators. AUs and gaze vectors are derived from the Facial Action Coding System (FACS), a comprehensive system developed to represent human facial movements. Detection of emotions is accomplished by employing a Convolutional Neural Network (CNN) that has been pre-trained on the AffectNet image database. A deep neural network model that combines these multiple modalities is finally used to estimate student engagement. Its performance is evaluated on the University of Massachusetts Amherst / Boston University student engagement dataset. The performance of the neural network model is characterized over a number of epochs to improve its bias-variance performance, thereby optimizing the model's overall accuracy. The performance of each modality is first explored. We then demonstrate the effectiveness of the multimodal approach, showcasing its potential. This paper first describes the current state of the art research into student engagement detection in e-learning settings, and concludes by outlining recommendations for future work in this up and coming area of study. This paper helps further our understanding of how suitable tools can be developed to detect student behaviors, and thereby ultimately improving the effectiveness of digital learning experiences in the context of remote learning.