Reinforcement Learning with Temporal Logic Specifications for Regression Testing NPCs in Video Games

Reinforcement learning (RL) is a promising strategy for the development of autonomous agents in various control and optimization contexts, including the generation of autonomous players in video games. However, designing these agents, and in particular their reward functions to perform sequential decision-making, can be challenging for most users and often require tedious trial-and-error processes until a satisfactory result is obtained. Consequently, these strategies are generally beyond reach for designers and quality control teams, who could potentially make use of them to generate automatic testing agents. This paper presents the application of reinforcement learning and behavioral descriptions given through a formal temporal logic task specification language (TLTL) for the design of NPCs that can be employed as surrogates for the player in such contexts. We argue that these techniques enable designers to naturally specify the way in which they would expect the final player to interact with a level and then generate a test that automatically verifies whether this strategy continues to be feasible throughout the development of the game. We include a series of experiments conducted on a custom 3D test environment developed in Unity3D that show that the proposed methodology provides a simple mechanism for training NPCs in settings that are commonly encountered in modern video games.