Real-Time Human Action Recognition With Dynamical Frame Processing via Modified ConvLSTM and BERT

In this study, a human action recognition approach with dynamical frame processing is proposed to fulfill the need for action recognition in a real-time manner. A novel architecture with a modified convolutional long short-term memory (ModConvLSTM) with pose heatmaps as input features is proposed to achieve human action recognition without a fixed frame number. The effects of ModConvLSTM are verified at different depths. In our proposed architecture, we replace global average pooling (GAP) with Bidirectional Encoder Representations from Transformers (BERT) to address the limitations of temporal processing in a two-dimensional convolutional neural network (2D-CNN). By incorporating a training-stage mask and leveraging BERT’s attention mechanism, the model gains improved contextual understanding. This enhancement enables our network to achieve accuracies of 91.46% on the NTU60 dataset and 83.06% on NTU120, marking an improvement of 1.63% and 3.22%, respectively, with only a slight increase of 0.1 GFLOPs in computational cost. Unlike other real-time models that rely on large-scale datasets such as Kinetics for extensive pretraining, our model achieves competitive performance while being trained directly on NTU datasets. A key contribution of our work is its ability to perform dynamical frame recognition, efficiently updating the ModConvLSTM cell and computing BERT every four new frames, achieving real-time inference in just 14.2ms on a CPU. This study enables continuous action recognition without requiring predefined action boundaries, making our model well-suited for real-world applications where actions occur naturally without explicit segmentation.