Multimodal Fusion for Human Action Recognition via Spatial Transformer

The recent generation of Microsoft Kinect cameras capture a range of multimodal signals providing RGB video, depth sequences, and skeleton information, making it an option to achieve enhanced HAR performance by fusing different data models. However, most existing fusion methods simply fuse multimodal features with a large amount of background noise, ignoring the underlying semantics between different models, resulting in a lack of accuracy. In this work, we propose a bilinear pooling and attention network (BPAN) fusion mechanism based on spatial transformers for human action recognition. This work improves the recognition accuracy in the following aspects.1) An effective two-stream feature pooling and fusion mechanism is proposed. RGB frames and skeletons are fused to enhance the spatiotemporal feature representation. 2) A spatial visual transformer is proposed and the framework is evaluated based on two widely used video action benchmarks, and the proposed method exhibits more favorable performance compared to the existing state-of-the-art methods.