Covariance Matrix Adaptation for Multi-Agent Proximal Policy Optimization

Pertinent works have shown that the on-policy reinforcement learning algorithm Multi-Agent Proximal Policy Optimization(MAPPO) can perform similarly to other off-policy learning algorithms in multi-agent systems through minimal hyperparameter tuning. However, previous literature has indicated that in multi-agent tasks, although refined agent features improve the utility of the algorithm, they inevitably affect efficiency. Meanwhile, the policy generated from sampled advantage value may lead to overfitting and the algorithm reduces the exploration variance prematurely which leading the policy falling into the local optimal solution. Drawing inspiration from Proximal Policy Optimization With Covariance Matrix Adaptation(PPO-CMA), we improve the update of the loss function in continuous action space. Experimental results show that our method is superior to the original MAPPO and QMIX algorithms.