Multimodal Adaptive Graph Convolutional Network for Behavior Recognition

Skeleton based graph convolutional behavior recognition algorithms are excellent in feature extraction and robustness, but still have limitations in dealing with subtle limb movements. To this end, we propose a multimodal data fusion-based adaptive graph convolution (MAGCN) behavior recognition method. Firstly, given the advantage that inertial sensors can accurately capture subtle movements without being affected by occlusion, the fusion topology graph is constructed by fusing the inertial data in the form of increasing nodes with the skeleton data through an early fusion strategy. Next, a gating mechanism is introduced into the graph convolutional layer to better learn the graph topology by dynamically adjusting the weights of the global and independent graphs. Finally, a channel attention mechanism is embedded in each spatio-temporal module to enhance the ability to capture key features. The method is evaluated on a publicly available dataset to validate its effectiveness.