Deep Reinforcement Learning Based Upper Limb Neuromusculoskeletal Simulator for Modelling Human Motor Control

The neuromusculoskeletal modeling and simulator (NMMS) have been widely utilized in various fields and applications. The deep reinforcement learning (DRL) algorithm is a promising method to study human motor controls and movement biomechanics via NMMS without experimental data. However, existing research lacks exploration of the DRL implementation for controlling neuromusculoskeletal simulators, and only a few have presented myoelectric control systems applied to the DRL-based NMMS. In this work, an off-policy DRL algorithm, Deep Deterministic Policy Gradient (DDPG), was implemented on an upper limb NMMS with two different types of action space - direct muscle activation output and PD-based internal model, and compared their control performance. In addition, we evaluated the performance of proportional myoelectric control systems implemented on the DRL-based upper limb NMMS. The results indicate that the DRL-based NMMS can execute upper limb movements accurately, and the proportional myoelectric control system reduced the muscle activation under both types of action space. Moreover, the PD-based internal model action space shows better learning and error-tracking performance than the direct muscle activation output action space.