Behavior Monitoring System Leveraging Human Pose Estimation

In recent years, human-centric computer vision technologies, such as human pose estimation and action recognition, have garnered significant attention. This study, therefore, proposes the use of a classroom behavior monitoring system that integrates off-the-shelf human pose estimators to compute behavioral features from the video data. The extracted features include continuous values (body movements, head angles, etc.) and discrete features (timing of hand-raising), which were not leveraged in prior studies. The system can operate in real-time on consumer-grade GPUs. This makes it useful for real-time feedback during classes and post-class analysis of video recordings. The proposed system was experimentally validated using a motion capture dataset featuring scenes with occlusions. We demonstrated its accuracy and real-time performance, along with the challenges of capturing full-body poses in complex environments. Additionally, we applied this system to a novel learning analytics task: estimating cognitive and non-cognitive abilities from videos using real-world data. As part of our analysis, we processed a video from an elementary school classroom with our system and linked the computed metrics to self-reported skills. These experiments demonstrated promising results, highlighting the potential of evaluating student skills solely based on visual cues.