Real-time classification of multi-modal sensory data for prosthetic hand control

Recent work on myoelectric prosthetic control has shown that the incorporation of accelerometry information along with surface electromyography (sEMG) has the potential of improving the performance and robustness of a prosthetic device by increasing the classification accuracy. In this study, we investigated whether myoelectric control could further benefit from the use of additional sensory modalities such as gyroscopes and magnetometers. We trained a multi-class linear discriminant analysis (LDA) classifier to discriminate between six hand grip patterns and used predictions to control a robotic prosthetic hand in real-time. We recorded initial training data by using a total number of 12 sEMG sensors, each of which integrated a 9 degree-of-freedom inertial measurement unit (IMU). For classification, four different decoding schemes were used; 1) sEMG and IMU from all sensors 2) sEMG from all sensors, 3) IMU from all sensors and, finally, 4) sEMG and IMU from a nearly optimal subset of sensors. These schemes were evaluated based on offline classification accuracy on the training data, as well as with task-related metrics such as completion rates and times for a pick-and-place real-time experiment. We found that the classifier trained with all the sensory modalities and sensors (condition 1) attained the best decoding performance by achieving a 90.4% completion rate with an average completion time of 41.9 sec in real-time experiments. We also found that classifiers incorporating sEMG and IMU information outperformed on average the ones that only used sEMG signals, even when the amount of sensors used was less than half in the former case. These results suggest that using extra modalities along with sEMG might be more beneficial than including additional sEMG sensors.