Ensemble critic network with risk estimation for multi-agent reinforcement learning

Reinforcement learning (RL) allows agents to optimize policies to maximize long-term cumulative rewards in an environment. However, the uncertainty embedded in the environment can lead to biased value estimation and potential risks. Especially in multi-agent systems, the increased complexity of interactions results in more widespread and difficult-to-quantify risk sources. Current risk-sensitive exploration principles for multi-agent scenarios often fail to meet the stability requirements of RL in high-dimensional complex spaces. To address these challenges, we propose the Ensemble Critic network with Risk Estimation for Agent Training framework (ECREAT), which enhances training stability and exploration safety. Focusing on the complex risk issues in multi-agent reinforcement learning (MARL), we introduce an ensemble critic algorithm based on the prediction of future Conditional Value at Risk (CVaR). By designing a novel Q-value update mechanism and multi-agent risk estimation metrics, ECREAT significantly improves safety and exploration efficiency in a wide range of complex tasks. The experimental results on the benchmark platforms indicate that ECREAT demonstrates outstanding overall performance in search tasks with both static and dynamic targets. Moreover, the occurrence of high-risk behaviors is significantly reduced, thereby validating its efficiency and practicality in real-world multi-agent scenarios.