Deep Reinforcement Learning for Low-Cost Humanoid Robot Soccer Players: Dynamic Skills and Efficient Transfer

This research focused on whether Deep Reinforcement Learning (Deep RL) can enable a cheap, small humanoid robot to develop complex and secure movement skills, which can then be integrated into intricate behavioral strategies in dynamic environments. The humanoid robot, equipped with 20 actuated joints, was trained using Deep RL to participate in a skewed one-versus-one (1v1) soccer match. Initially, we taught the robot to separate skills before combining them in a self-play scenario. The final policy demonstrated robust and dynamic movement characteristics such as rapid fall recovery, walking, turning, kicking, and more, exceeding the logical expectations for the robot. The robot seamlessly transitioned between these skills in a stable and effective manner. Moreover, the agents developed an understanding of the game’s strategy, including predicting ball trajectories and countering opponent moves. Surprisingly, a limited number of straightforward rewards led to a wide range of behaviors. The agents underwent simulated training before being transferred to real robots. Despite unmodeled effects and differences between robot instances, successful transfer was assisted by using a combination of high-voltage control, concentrated dynamics selection, and training with disturbances in simulation. By introducing small hardware modifications and applying basic behavior regularization during training, the robots achieved secure maneuvers while maintaining dynamic and agile performance. Notably, the agents in the experiments demonstrated significant improvements over a programmed baseline, walking 156% faster, standing up 63% faster, and kicking 24% quicker, all while effectively combining their skills to achieve long-term goals.