EchoGest: A Highly Scalable Unseen Gesture Recognition System Based on Feature-Wise Transformation

Recent research studies have made significant progress in acoustic-based gesture recognition. However, the existing methods lack the capability to expand to customized gestures and adapt to different practical environments. We propose a highly scalable gesture recognition system called EchoGest which integrates a well-designed feature-wise transformation layer into the prototypical network framework, and accomplishes unseen gesture recognition with a device’s built-in speaker and microphone. Our key insight involves gauging the similarity between the query sample representations and class prototypes in the embedding space, and thus enabling the scalability to unseen gestures. Meanwhile, we introduce a feature transformation layer to linearly adjust the feature maps and propose an efficient two-stage training strategy to obtain regularized parameters for this layer. Specifically, this layer employs affine transformation to enhance intermediate feature activations and yield more diverse feature distributions for cross-domain recognition, and it improves recognition accuracy by 10% in one-shot cases. We train the system with a collected a letter gestures (i.e., writing “A” to “Z”) data set and test it on a digit gestures (i.e., writing “0” to “9”) data set with ten volunteers. The results show that the EchoGest can recognize unseen digit gestures with an accuracy of 93.7% in two-shot cases, and 93.2% in the leave-one-user-out testing setting. We also explore a semi-supervised clustering approach in which each user’s data can be used to update his or her prototypes for personalized customization. The comprehensive experiments also verify that the EchoGest remain good performance across various environments, age groups, and different devices.