Applying Deep Reinforcement Learning to Train AI Agents in a Wargaming Framework

The United States Navy's mission planning process consists of six steps. The third step, referred to as the course of action (COA) analysis (wargaming) process, encompasses the evaluation of initial actions taken, the corresponding reactions by opponents, and the associated counteractions. This entire process supports developing potential mission COAs while also informing decision makers of potential outcomes in support of planning and preparing for operations. This wargaming process may occur using table-top exercises with subject-matter experts that determine actions taken. However, continued improvements with deep reinforcement learning (DRL), a sub-field of machine learning, provide an opportunity to leverage the use of artificial intelligence (AI) agents within the wargaming process. The AI agents must perform credibly to represent a believable behavior of either the red team or blue team. Developing credible behavior requires tailoring agent reward functions, analyzing impacts of different training algorithms and parameter values, and understanding and evaluating the resulting behavior. This work also demonstrates agent behavior using a browser-based gameplay interface accessible to decision makers. This paper provides an overview of research efforts to train and analyze AI agent performance using DRL within a prototype government owned wargaming framework, discusses the capabilities and utility of the browser-based interface, and explores challenges and opportunities for future research.