RoMA: Resilient Multi-Agent Reinforcement Learning with Dynamic Participating Agents

This paper presents RoMA, a novel resilient Multi-Agent Reinforcement Learning (MARL) framework designed to handle dynamic participating agents during centralized training, addressing the limitations of standard MARL frameworks in accommodating agent variability and enabling efficient adaptation and training of agents, thus providing a scalable and flexible solution for model training and execution in cloud computing environments. For standard MARL frameworks, if new agents need to join or existing agents leave unexpectedly due to unreliable communication channels, standard MARL models need to be rebuilt and trained from scratch because of their structural limitations, which is very time-consuming. RoMA addresses this issue with a novel neural network architecture and a few-shot learning algorithm to enable the number of agents to vary during centralized training. When new agents join, RoMA can adapt all agents to the change in a few shots, and when agents leave the training process unexpectedly, RoMA can continue training the remaining agents without disruption.Our experiments demonstrate that RoMA is at least 70 times faster at adapting to new agents compared to baseline methods, and it can handle the leaving of agents without affecting the training of other agents. RoMA is applicable to a wide range of MARL settings, including cooperative, competitive, independent, and mixed environments.