FI-HGR: A Robust Hand Gesture Recognition System Based on Wearable Data Glove and Multimodal Fusion Algorithm

Hand gesture recognition (HGR) plays a crucial role in human-computer interaction systems within the Internet of Things (IoT). Recent HGR methods often rely on vision-based images or videos, which are limited in terms of occluded fingers and high computational cost due to complex neural networks. In contrast, wearable sensors like inertial measurement units (IMUs) and flexible sensors can handle hand self-obscuration. However, there are two unresolved issues. First, using a single modality is hard to balance high precision and low latency. Second, existing multimodal-based approaches lack deep inter-modal coupling to effectively address IMU drift and mechanical coupling of flexible sensors. To address these problems, we propose FI-HGR (HGR based on flexible and inertial data). FI-HGR comprises a sensor-integrated data glove and a novel Cascaded Complementary-Stochastic Fusion Algorithm (CS-Algorithm). CS-Algorithm employs six Mahony filters to estimate the state quaternion of each IMU, along with an Extended Kalman Filter that continuously corrects IMU drift based on the index finger’s bending angle sensed by a flexible sensor. This design allows a single flexible sensor to calibrate multiple IMUs and introduces a hard constraint, resolving the sensor drift problems that previous methods cannot. Based on the CS-Algorithm outputs, precise finger bending angles are estimated in real time. Subjective and objective experimental results show that our approach effectively addresses IMU drift and mechanical coupling in flexible sensors, reduces gesture tracking error to approximately 3.4∘, and significantly improves both recognition accuracy and operational efficiency.