A Two-Stage Modality Fusion Approach for Recognizing Human Actions

Human action recognition (HAR) task, as one of the most representative human-centric visual understanding tasks, has received extensive attention from academia and industry. The key issue of this task is to mine the feature representations that can effectively describe the characteristics of human actions. In this article, we make full use of the information from multiple modalities in video, and propose a two-stage modality fusion strategy, which effectively mitigates the modality gaps between different modalities. First, in the first-stage fusion, based on the two-stream network, homogeneous RGB image’s and optical flow’s representations are aligned and integrated through generative adversarial network (GAN) and deep belief networks (DBNs), respectively. Then, in the second-stage fusion, the features obtained in the first stage are fused with the heterogeneous skeleton features through the proposed adaptive bilinear pooling network. Finally, the fused discriminative features are input into the classifier for recognition. Through this two-stage fusion strategy, the consistent and complementary representations between different modalities are fully mined. Experimental results show that our proposed approach outperforms several other methods on NTU-RGB + D (CS 94.5%, CV 98.8%), NTU-RGB + D 120 (CS 93.4%, CSet 94.8%), Northwestern UCLA (
${V}_{\text {1,2}}^{{3}} 97.2$
%), and Penn Action datasets (Top1 99.8%).