A Real-Time Cross-Domain Wi-Fi-Based Gesture Recognition System for Digital Twins

The rapid development of Internet of Things has led more realization of digital twins (DT), such as healthcare, smart homes, virtual reality, etc., gesture recognition is a fundamental component of DT. Its implementation can provide users with personalized services or improved human-computer interaction, such as smart home control, in-car interaction, etc., most of existing gesture recognition methods are based on vision or wearable device. However, the vision-based methods face the problem of privacy breach, whereas the wearable-based methods may bring inconvenience to users. With the wide deployment of Wi-Fi networks, lots of consumer devices are widely accessible in people’s homes. Motivated by the fact that Wi-Fi signal propagation can be affected by human motion, the opportunity to use Wi-Fi signals for gesture recognition can be further explored. However, the challenge is that the received Wi-Fi signal shows great differences when the same person performs the same gesture in different environments or different person performs the same gesture in the same environment. Therefore, the signal alignment across different domain needs to be solved. In this paper, we propose a gesture recognition system named Phase-Attention-based-Conv-CSI (PAC-CSI), which consists of two modules: data processing and gesture recognition. In the data processing module, we eliminate random phase noise in channel state information (CSI) and perform phase calibration. In the gesture recognition module, we feed the processed phase sequence into a lightweight deep neural network for gesture recognition. PAC-CSI can obtain the gesture category in about 200ms, which can meets the real-time requirements of DT. The gesture recognition accuracy of our proposed system in a single domain is 99.46%, and its performance across new locations, orientations, users, and environments is 98.77%, 98.90%, 97.54%, and 96.47%, respectively.