A Novel sEMG-FMG Combined Sensor Fusion Approach Based on an Attention-Driven CNN for Dynamic Hand Gesture Recognition

Surface-electromyogram-based pattern recognition (sEMG-PR) is considered as a promising intuitive control method for multifunctional prostheses. However, sEMG-PR relies on the unreliable assumption that repeatable muscular contractions produce repeatable patterns of steady-state sEMG. In contrast, the transient-state signal associated with the beginning (onset) of muscle contraction contains substantial temporal information useful for motor intention characterization but has rarely been explored. In this study, we proposed a cross-attention convolutional neural network (CNN-ATT) that fused sEMG and force myography (FMG) transient signals for multiclass dynamic gesture characterization. The effectiveness of the proposed model was validated using a self-developed co-located system for simultaneously acquiring sEMG and FMG recordings from ten subjects who performed 15 hand gestures. The result showed that the FMG signal performed better than its sEMG counterpart with a performance improvement of 9%, while the CNN-ATT result demonstrated classification performance of 96%, which is 12% higher than sEMG alone and 3.3% higher than FMG alone. To the best of our knowledge, this study represents the first to explore the combination of sEMG and FMG signals for hand gesture recognition based on transient sEMG signals. The results of this study may provide a novel and efficient method for dynamic control of not only intelligent prosthetic hands but also gaming and rehabilitation systems.