Continuous Hand Gestures Detection and Recognition in Emergency Human-Robot Interaction Based on the Inertial Measurement Unit

Robots have become increasingly prevalent in various aspects of daily life. In an increasing number of scenarios, particularly in social contexts, such as those involving delivery robots and self-driving buses, human-robot interaction (HRI) is essential for conveying decision commands to the robot’s control algorithm. The growing convergence of human activity spaces and robot workspaces presents increasingly emergent and challenging interaction scenarios. This transformation necessitates convenient measurement sensor and gesture recognition algorithms with shorter recognition delays and more reliable recognition results. In pursuit of this objective, we propose a novel pipeline for continuous and rapid gesture recognition employing a portable inertial measurement unit (IMU) device. First, we design a lightweight neural network (NN) to distinguish the gesture from daily hand movement (gesture spotting). Then, one lookback window evaluation is inserted between gesture spotting and recognition, aiming to reduce the false positive rate of gesture spotting. Finally, a novel transformer-based model is used to recognize expected hand gestures (gesture recognition), which can consider both cross-time and cross-sensor dependencies. In this study, the IMU data of six gestures are collected from 20 participants for training and validation. The experimental results show that the proposed gesture spotting model detects all gesture movements in the test data with a 0.3% false positive rate. The gesture recognition model achieves better accuracy than multiple state-of-the-art models. Overall, this study proposes an efficient and reliable pipeline for single IMU-based continuous gesture recognition for practical applications.