A Q-learning Novelty Search Strategy for Evaluating Robustness of Deep Reinforcement Learning in Open-world Environments

Despite substantial progress in deep reinforcement learning (DRL), a systematic characterization of DRL agents’ robustness to unexpected events in the environment is relatively under-studied. Such unexpected events (“novelties”), especially those that are more structural than parametric, may significantly deteriorate the performance of DRL agents, leading them to be unfit for open-world environments and applications. However, not all novelties affect an agent’s performance equally. Unfortunately, even with reasonable and constrained definitions of the problem, the space of all novelties can be (at least) exponential. Hence, an effective search strategy is required to find novelties that can adversely affect the agent. This paper presents a formalism for this problem and proposes a deep Q-learning-based novelty-search strategy that efficiently and systematically finds candidate (potentially complex) novelties with significant negative impact on a DRL agent. We conduct a detailed set of experiments in a stochastic multi-agent game environment (Monopoly) with complex decision-making properties.