Improving Unimodal sEMG-Based Pattern Recognition Through Multimodal Generative Adversarial Learning

Surface electromyography (sEMG)-based pattern recognition system is a kind of instrumentation and measurement system that captures, processes, and recognizes bioelectrical signals, like other pattern recognition systems, it can benefit from multimodal data to enhance its recognition performance. However, acquiring multimodal data typically requires additional sensors, thereby increasing hardware complexity and costs. This article proposes a novel generative auxiliary modality (GAM) method. It improves the overall performance of unimodal sEMG-based hand gesture recognition (HGR) by integrating a multimodal generative adversarial learning process into the HGR model training phase. Specifically, we trained a deep generative model using real sEMG and inertial measurement unit (IMU) signals, which is able to generate corresponding virtual IMU signals from the input real sEMG signals. During gesture recognition, the generated virtual IMU signals were used as the auxiliary modality alongside sEMG signals to provide multimodal information for unimodal sEMG-based gesture recognition, both of which are subsequently fed into a multimodal convolutional neural network (CNN) for gesture recognition. Evaluations were conducted on six databases, including five publicly available databases and our collected database containing sEMG and IMU data from 28 subjects performing 38 gestures. Results demonstrate that by incorporating generated virtual IMU signals, our method can significantly improve unimodal sEMG-based HGR accuracy (with increases of 2.15%–11.43% and 2.54%–8.29% in intrasubject and intersubject HGR accuracy, respectively) while only incurring an acceptable increase in computational cost. This improvement is particularly evident in distinguishing gestures with identical hand gestures across different forearm postures. Moreover, the accuracy of our proposed GAM method is comparable to multimodal HGR systems using both real sEMG and IMU signals as their inputs. This work represents a successful attempt to provide auxiliary modal signals for sEMG-based HGR through multimodal generative adversarial learning without the need for physical IMU sensors, thus having the potential to reduce hardware complexity and costs.