Heterogeneous Multiagent Zero-Shot Coordination by Coevolution

Generating agents that can achieve zero-shot coordination (ZSC) with unseen partners is a new challenge in cooperative multiagent reinforcement learning (MARL). Recently, some studies have made progress in ZSC by exposing the agents to diverse partners during the training process. They usually involve self-play when training the partners, implicitly assuming that the tasks are homogeneous. However, many real-world tasks are heterogeneous, and hence previous methods may be inefficient. In this article, we study the heterogeneous ZSC problem for the first time and propose a general method based on coevolution, which coevolves two populations of agents and partners through three subprocesses: 1) pairing; 2) updating; and 3) selection. Experimental results on various heterogeneous tasks highlight the necessity of considering the heterogeneous setting and demonstrate that our proposed method is a promising solution for heterogeneous ZSC tasks. To the best of our knowledge, we are the first to underscore the significance of the heterogeneous ZSC tasks and to introduce an effective framework for addressing it.