A Novel Multimodal Human Activity Recognition based on Self-Attention Mechanism

Multimodal human activity recognition attracts wide attention in human-computer interaction. However, in the collected multimodal signals, not all modal signals contain useful feature information; some irrelevant and redundant information may negatively impact the model’s performance, reducing the accuracy of activity recognition. This paper designs a self-attention mechanism-based multimodal fusion network for combining $\mathbf{W i}$ Fi signals and image streams based on video signals. The self-attention mechanism possesses the capability to capture spatiotemporal local features within multimodal signals. It dynamically learns the weights of different modalities, assigning higher weights to relatively important modalities. This process effectively fuses features extracted from individual modalities, resulting in a more comprehensive feature set. Through extensive experiments, we evaluate the performance of the proposed multimodal human activity recognition network from various perspectives. The experimental results indicate the effectiveness of the proposed method for data collected in different scenarios.