Analysis of Classroom Processes Based on Deep Learning With Video and Audio Features

An active learning-type class is a class in which students take the initiative. In order to improve active learning-type classes, attempts have been made to review the content after the class using video taken by the lecturer, but this is burdensome because it takes a long time to review the video. Although methods have been proposed for estimating the classroom process at each time, there is still room for improvement. In this paper, we propose a method for estimating the classroom process at each time with higher accuracy than conventional methods. The proposed method uses deep learning to improve the accuracy of the conventional method, which only uses classical SVM. We also used an ablation study to find an appropriate combination of input modalities. Furthermore, we introduced ensemble LSTM to handle data with different modalities. The proposed method achieved the highest accuracy of 98.8% and the lowest accuracy of 64.4% in estimating classroom activities, with an average classification accuracy of 80.1%.