DAMUN: A Domain Adaptive Human Activity Recognition Network Based on Multimodal Feature Fusion

There is a rapidly increasing demand for human activity recognition (HAR) due to its extensive applications in various fields such as smart homes, healthcare, nursing, and sports. A more stable and powerful system that can adapt to various complex actual environments with affordable cost of data acquisition is needed. In this article, we propose a domain adaptive HAR network based on multimodal feature fusion (DAMUN) to capture information of data from frequency-modulated continuous wave (FMCW) radar and USB cameras. In the network, we add a domain discriminator to reduce data differences due to the changes in environments and user habits. In order to reduce the workload of radar data acquisition and processing, we also design a data augmentation model based on a generative adversarial network (GAN), which can generate radar data directly from image data. Finally, we implement the real-time application based on the DAMUN on edge computing platforms. The experimental results show that the proposed network achieves obvious advantages over the existing methods and can effectively adapt to different environments. In addition, the network can meet the real-time requirement in the prediction stage, and its average running time is about 0.17 s.