Simulation Control of Hierarchical PPO Reinforcement Learning for Heavy-Duty Hexapod Robots in Complex Terrain

To solve the problems of intermittent ground contact and limited dynamic terrain adaptability in heavy-duty hexapod robots. A reinforcement learning control framework based on hierarchical Proximal Policy Optimization is proposed, the dual layer decision-making architecture is adopted by this framework. The high-level strategic network is gradually adapted to terrains of different complexities through course learning, the global gait parameters are dynamically generated such as stride and stride frequency. The GAE advantage estimation is adopted by the low-level strategy network to accurately control the torque of each joint. A multi-objective reward function is designed to improve the task completion rate, stability and movement efficiency. Four types of simulated terrain are constructed for simulation experiments, the results show that the average reward value of this method in complex terrain is 0.15-0.5 higher than standard PPO, and the training convergence speed is improved by more than 0.4. Finally, the changes of roll pitch yaw angle of the hexapod robot in the continuous undulating terrain(a) is taken as an example, the stability and robustness of hierarchical PPO reinforcement learning algorithm in the control of hexapod robot are verified.