Hierarchical Reinforcement Learning with Partner Modeling for Distributed Multi-agent Cooperation

Many real-world applications can be formulated as multi-agent cooperation problems, such as network packet routing and coordination of autonomous vehicles. The emergence of deep reinforcement learning (DRL) provides a promising approach for multi-agent cooperation through the interaction of the agents and environments. However, traditional DRL solutions suffer from the high dimensions of multiple agents with continuous action space during policy search. Besides, the dynamicity of agents' policies makes the training non-stationary. To tackle these issues, we propose HERO, a distributed hierarchical reinforcement learning approach that reduces training time and improves model performance by decomposing the overall policy into multiple layers of sub-policies in a hierarchical manner. On the one hand, the cooperation of multiple agents can be learned efficiently in high-layer discrete action space. On the other hand, the low-layer individual control can be reduced to single-agent reinforcement learning. Furthermore, we introduce a partner modeling network to effectively capture and model other agents' policies during learning. Finally, We develop a real-world multi-vehicle testbed and evaluate HERO in a series of cooperative lane change scenarios. Both the simulation and real-world experiments show that our approach can achieve a lower average collision rate of 0.09 and a faster convergence speed than other baselines.