Expression Recognition and Intelligent Classroom State Mining in Teaching Videos Based on Semi-Supervised Learning Generative Adversarial Network

We constructed a student classroom teaching video dataset, constructed a student classroom expression dataset through video frame extraction, face detection, face alignment, and facial extraction, and constructed a student classroom behavior dataset through object detection algorithm. Aiming at the characteristics that the annotation information of the students' classroom expression dataset constructed in this paper is limited and the data of some categories are less, an expression recognition method based on semi-supervised learning generative adversarial network (SSLGAN) is proposed. The problem of limited annotation information is solved through semi-supervised learning, and the problem of less data of some categories is solved through data enhancement of generative adversarial network. An intelligent classroom state mining method that integrates student expressions and behaviors has been proposed, achieving intelligent classroom state mining of teaching videos and automatic evaluation of classroom listening status for individual and overall students in different types of classrooms, such as interactive classes, discussion classes, and self-study classes. Compared with CNN, RMN, Temporal Ensembling and Mean Teacher, the proposed SSLGAN has achieved the highest accuracy in the benchmark dataset FER2013, JAFFE, RaFD and self-built student classroom expression dataset. In the case where there are few labels in the self-built student classroom expression dataset, the accuracy of student expression recognition results still reaches 76%, and the proposed intelligent classroom state mining method that integrates student expressions and behaviors for different types of classrooms also has high accuracy on the self-built student classroom teaching video dataset.