Learning Diverse Control Strategies for Simulated Humanoid Combat via Motion Imitation and Multi-Agent Self-Play

Human athletes can coordinate their bodies to achieve exquisite strategies with agile and diverse motions in multiplayer competitions, which is a long-standing challenge for robotics and physically embodied artificial agents. In this paper, we propose a hierarchical learning framework that generates diverse control strategies and agile motion for physically simulated humanoids, which can be divided into two stages: learning basic motion skills and learning high-level competitive strategies. The framework decouples low-level control and high-level strategy learning, where the low-level policy is trained via motion imitation and skill discovery objectives to generate agile motion skills, and the high-level policy is trained via prioritized fictitious self-play to generate diverse competitive strategies. Furthermore, we develop and release the world's first physically simulated humanoid combat environment. We evaluate our learning framework in this environment, and experimental results demonstrate that policies learned by our framework can generate both agile motion skills and diverse long-term competitive strategies.