Electromyography-Based Decoding of Dexterous, In-Hand Manipulation of Objects: Comparing Task Execution in Real World and Virtual Reality

The increased use of Virtual and Augmented Reality based systems necessitates the development of more intuitive and unobtrusive means of interfacing. Over the last years, Electromyography (EMG) based interfaces have been employed for interaction with robotic and computer applications, but no studies have been carried out to investigate the continuous decoding of the effects of human motion (e.g., manipulated object behavior) in simulated and virtual environments. In this work, we compare the object motion decoding accuracy of an EMG based learning framework for two different dexterous manipulation scenarios: i) for simulated objects handled by a teleoperated model of a hand within a virtual environment and ii) for real, everyday life objects manipulated by the human hand. To do that, we utilize EMG activations from 16 muscle sites (9 on the hand and 7 on the forearm). The object motion decoding is formulated as a regression problem using the Random Forests methodology. A 5-fold cross validation procedure is used for model assessment purposes and the feature variable importance values are calculated for each model. The decoding accuracy for the real world is considerably higher than the virtual world. Each of the objects examined had a single manipulation motion that offered the highest estimation accuracy across both worlds. This study also shows that it is feasible to decode the object motions using just the myoelectric activations of the muscles of the forearm and the hand. This is particularly surprising since simulations lacked haptic feedback and the ability to account for other dynamic phenomena like friction and contact rolling.