WVGR: Gesture Recognition based on WiFi-Video Fusion

Gesture recognition is widely used in emerging industries, such as smart homes, smart medical care, and human-computer interaction. Despite its promising applications, gesture recognition based on a single-modal, such as WiFi or Video, has significant limitations. WiFi-based gesture recognition suffers from the susceptibility of its signals to multipath interference, affecting the sensing data quality, while Video signals are light and shadow sensitive and cannot penetrate obstacles, which limits its practicality in gesture recognition, particularly in cross-target recognition settings. To overcome these limitations, this paper proposes a WiFi-Video based gesture recognition method WVGR. WVGR utilizes both contrastive and iterative learning to combine the useful features of WiFi and Video, thereby increasing the accuracy of cross-target gesture recognition. Specifically, WVGR develops a two-stage learning framework. The first stage involves the fusion of WiFi and Video data utilizing a contrastive learning-based method, while the second stage integrates few labeled data to improve the efficiency of the fusion process. We recruited 10 volunteers to participate in the experiment, including two modalities of WiFi and Video, 6 gestures, and each gesture was repeated 30 times. Experimental results demonstrate that WVGR improves the recognition performance of cross-target gesture recognition. In particular, WVGR achieved an accuracy improvement of 18.8% over WiFi-based recognition and 4.4% over Video-based recognition.