Analysis of Students' Positive Emotion and Smile Intensity Using Sequence-Relative Key-Frame Labeling and Deep-Asymmetric Convolutional Neural Network

Positive emotional experiences can improve learning efficiency and cognitive ability, stimulate students' interest in learning, and improve teacher-student relationships. However, positive emotions in the classroom are primarily identified through teachers' observations and postclass questionnaires or interviews. The expression intensity of students, which is extremely important for fine-grained emotion analysis, is not considered. Hence, a novel method based on smile intensity estimation using sequence-relative key-frame labeling is presented. This method aims to recognize the positive emotion levels of a student in an end-to-end framework. First, the intensity label is generated robustly for each frame in the expression sequence based on the relative key frames to address the lack of annotations for smile intensity. Then, a deep-asymmetric convolutional neural network learns the expression model through dual neural networks, to enhance the stability of the network model and avoid the extreme attention region learned. Further, dual neural networks and the dual attention mechanism are integrated using the intensity label based on the relative key frames as the supervised information. Thus, diverse features are effectively extracted and subtle appearance differences between different smiles are perceived based on different perspectives. Finally, comparative experiments for the convergence speed, model-training parameters, confusion matrix, and classification probability are performed. The proposed method was applied to a real classroom scene to analyze the emotions of students. Numerous experiments validated that the proposed method is promising for analyzing the differences in the positive emotion of students while learning in a classroom.