Entropy Maximization in High Dimensional Multiagent State Spaces

Underwater or planetary exploration are prime examples of missions that can benefit from autonomous agents working together. However, discovering effective team-level behaviors (i.e., coordinated joint actions) is challenging in these domains as agents typically receive a sparse reward (zero-or constant-for the majority of the interactions). To address this issue, intrinsic rewards encourage agents to explore diverse policies to visit the state space more effectively. Unfortunately, as the agents’ state space grows, intrinsic reward-based (i.e., curiosity) approaches become less effective as they cannot effectively distinguish a diverse set of states. In this direction, we introduce state entropy maximization for multiagent learning where agents explore using local (dense) rewards and learn to solve the coordination task by leveraging global (sparse) rewards. Because of the intrinsic ability to balance local and global rewards, our approach enables the state entropy function to remain effective in high dimensional state spaces. Experiments in tightly coupled tasks requiring complex joint actions, show that local entropy-based rewards enable agents to discover successful team behaviors in high dimensional spaces where previous hand-tuned count-based rewards fail.