Neural Manifold Modulated Continual Reinforcement Learning for Musculoskeletal Robots

The continual learning and development are significant for robots to learn multiple tasks sequentially. The difficulty lies in balancing the efficient learning of new tasks and overcoming catastrophic forgetting of old tasks. Although many continual learning methods have been proposed for pattern recognition, continual reinforcement learning methods for redundant musculoskeletal and robotic systems are few and have limitations. Therefore, inspired by the developmental mechanisms in motor cortex, this article proposes a neural manifold modulated continual reinforcement learning method for musculoskeletal and robotic systems. First, a recurrent neural network (RNN) with an expected neural manifold is designed and conditions of weights are derived. Second, the ability of projectors for characterizing the neural manifold within RNN is analyzed. Furthermore, a continual reinforcement learning method of RNN is proposed with the modulation of a neural manifold. The method is validated by redundant musculoskeletal and robotic systems in simulation. The results suggest that it can realize continual reinforcement learning of multiple tasks in different movements and environments. Furthermore, compared with related works, the proposed method achieves better performance.