Class Activation Regularization-Based Facial Emotion Recognition Network and its Application in Students’ Emotional Engagement Assessment

Students’ emotional engagement in the classroom can be evaluated through their facial emotions. However, facial emotion recognition remains challenging due to the uncertainty problem, which is routinely caused by factors such as low resolution, intra-class variability, and inter-class similarity. To deal with this problem, we propose a class activation regularization network, which includes two main modules: class activation dropping module (CADM) and hard sample mining module (HSMM). Our key idea is to perform consistency learning between the original images and class-uncertain regions, thereby enabling models to perceive uncertainty of emotional semantics. In CADM, the class-specific attention is obtained using class activation mapping. Then, the attention is partly discarded by imitating dropout mechanism to obtain class-uncertain regions. In HSMM, the hard samples with high uncertainties are identified based on the class-uncertain regions and confidence thresholds. For the hard samples, regularization loss is applied to align the probability distributions between the original images and the class-uncertain regions, thereby improving the network’s awareness of emotional semantics and its ability to handle uncertainty. Extensive experiments on three datasets demonstrate the competitiveness of the proposed method. In addition, our method can be deployed in the classroom for students’ emotional engagement assessment.