Prediction-based Hierarchical Reinforcement Learning for Robot Soccer

Robot soccer as a complex mixed cooperative-competitive task presents many challenges to multi-agent reinforcement learning (MARL), such as assigning long-term credits and effective exploration in high-dimensional and continuous state-action spaces. We propose Prediction-based Hierarchical Reinforcement Learning (P-HRL) for robot soccer. P-HRL consists of a coach for soccer tactics and a robot controller for robot motion control. To comprehensively evaluate the performance of P-HRL, we design various key performance indicators for robot soccer such as ball possession rate. Experimental results demonstrate that P-HRL has a better performance than the baseline MATD3, with 52% win rate, 22% draw rate, and 26% loss rate.