Automatic Action Space Specification for Deep Reinforcement Learning in Games via Program Analysis

Reinforcement learning has numerous applications for the testing and analysis of video game content. However, deploying it in existing games is a challenging engineering effort, requiring suitable representations of states, actions, and rewards based on the game rules. We propose using the program analysis technique of symbolic execution on the game code to automatically determine a precise action model when deploying reinforcement learning in existing games. Our technique automatically computes appropriate discrete action spaces for games, including action masks indicating the validity of actions depending on the agent's current state. We conduct a comprehensive evaluation of the technique on a varied dataset of seven Unity games with the Proximal Policy Optimization (PPO) and Deep Q Learning (DQN) deep reinforcement learning approaches. The results show that the agents using the analysis significantly out-perform those using generic action spaces covering the input device, and perform on par with those using manually specified action spaces.