Video Traffic Analysis for Real-Time Emotion Recognition and Visualization in Online Learning

Since the outbreak of the COVID-19 crisis, the transition to remote education presented several challenges to educational institutions. Unlike face-to-face classes where educators can modify and keep track of the lessons and content according to the students’ observed emotions and participation, such activities are difficult to complete in online learning environments. To address this issue, we propose here a novel and comprehensive framework that leverages advanced computer vision and analysis techniques to detect students’ emotions during online learning and assess their state of mind regarding the taught content. Our framework is composed of three modules. The first module uses a novel lightweight machine learning method, called convolutional neural network-random forest (CNN-RF), to efficiently detect the students’ basic emotions, e.g., sad, happy, etc., during the online course. Our approach surpasses existing benchmarks in terms of accuracy (over 71%) on the FER-2013 dataset, while being less complex (i.e., using a smaller number of parameters). The second module consists of mapping the basic emotions to an education-aware state of mind, e.g., interest, boredom, distraction, etc. Unlike the few works that proposed simplistic mapping, we propose here a Plutchik wheel’s inspired mapping system, which is more precise and reflects better the relationship between combinations of basic emotions and the resulting education-aware state of mind. Thus, our understanding of the students’ cognitive and affective experiences during online learning can be enhanced. The third module is a visualization dashboard that offers clear and intuitive real-time representations of basic emotions and states of mind. This tool provides educators with invaluable insights into students’ emotional dynamics, enabling them to identify learning difficulties with high precision and make informed recommendations for improvements in course content and online teaching methods. In summary, the proposed framework presents a novel and powerful tool that addresses the challenges related to online learning. By accurately detecting the students’ emotions, assessing their states of mind, and providing real-time visualization, our approach represents a significant advancement toward the optimization of online education, which is critically needed in rural and remote areas of the globe.