Large Scale Deep Reinforcement Learning in War-games

War-game is a type of multi-agent real-time strategy game, with challenges of the large-scale decision-making space and the flexible and changeable battlefield situation. In addition to the military field, it has played a role in fields including epidemic prevention and pest control. In recent years, more and more learning algorithms have tried to solve this kind of game. However, the existing methods have not yet given a satisfactory solution for the war-game, especially when preparation time is limited. In this background, we try to solve a traditional war-game based on hexagon grids. We propose a hierarchical multi-agent reinforcement learning framework to rapidly training an AI model for the war-game. The higher-level network in our hierarchical framework is used for task decision, it solves the credit assignment problem between agents through cooperative training. The lower-level network is mainly used for route planning, and it can be reused through parameter sharing for all the agents and all the maps. To deal with various opponents, we improve the robustness of the model through a grouped self-play approach. In experiments, we get encouraging results which show that the hierarchical structure allows agents to learn their strategies effectively. Our final AI model demonstrates that our methods can effectively deal with the challenges in the war-game.