An AIoT Framework With Multimodal Frequency Fusion for WiFi-Based Coarse and Fine Activity Recognition

Benefiting from the progresses of sensing and sustainable computing technologies, recent years have witnessed the dramatic progresses of artificial intelligence of Things (AIoT). As a typical AIoT application, WiFi-based human activity recognition has increasing popularities in smart homes. However, WiFi-based action recognition often has unstable performance due to environmental interference. To this end, a robust deep learning framework called multiple spectrogram fusion network (MSF-Net) is proposed for coarse and fine activity recognition using channel state information (CSI). First, a dual-stream structure incorporating short-time Fourier transform and discrete wavelet transform is developed to highlight abnormal information in the CSI data. Then, a Transformer is employed as the backbone to effectively extract high-level features. In addition, an attention-based fusion branch is designed to enhance cross-model fusion. Experimental results show that the MSF-Net achieves Cohen’s Kappa scores of 91.82%, 69.76%, 85.91%, and 75.66% on the SignFi, Widar3.0, UT-HAR, and NTU-HAR data sets, respectively. These performance records demonstrate the advantages of MSF-Net over existing methods for coarse and fine activity recognition based on the WiFi data.