Graph-based Selection-Activation Reinforcement Learning for Heterogenous Multi-agent Collaboration

In practical applications of multi-agent systems, agents are often heterogeneous, and each type of them typically has different task objectives. For heterogeneous multi-agent reinforcement learning (HMARL), the diversity of agent types and the unbalanced agent number of each type can lead to the curse of dimensionality and non-stationary. Moreover, the increase in the number of heterogeneous agents may result in slow convergence during training. This paper proposes a graph-based selection-activation reinforcement learning (GSARL) method for training heterogenous multi-agent collaboration strategies. It first constructs agents based on their types, then extracts the global adjacency matrices and the ally adjacency matrices from the agents’ observations, and calculates the global feature matrices. Afterwards, GSARL utilizes hierarchical graph convolutional network to sequentially convolve the global information and ally information, obtaining action logits based on agent types. By using the neural topology graph and the selection-activation method, the optimal multi-agent collaboration configuration is obtained through combinatorial optimization. Experiments are conducted in an adversarial combat simulation environment involving collaborative Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs). Simulation results show that the proposed method can accelerate convergence while allowing that each type of heterogeneous agents can leverage its unique advantages.