A Visual Analytics Approach to Understanding Proximal Policy Optimization

PPO(Proximal Policy Optimization) is one of the most widely used and effective reinforcement learning algorithms, with successes in a variety of domains. However, understanding and diagnosing PPO can be challenging. In this work, we propose an interactive visual analytic system called PPOViz that helps users better understand and diagnose the PPO algorithm. PPOViz not only provides better visualization of the agent’s behavior, but also uses the Grad-CAM technique to make the agent’s model more interpretable. In addition, the gradient analysis of loss allows users to obtain other more useful information from loss. To demonstrate the effectiveness of PPOViz, we use the SpaceInvaders environment in Atari games to show how PPOViz helps users better understand and diagnose ways to improve the PPO algorithm.