Adversarial Testing with Reinforcement Learning: A Case Study on Autonomous Driving

Testing autonomous driving systems (ADSs) is essential to ensure their safety. Existing testing techniques manipulate the objects of the driving environment in order to trigger a misbehavior of the ADS under test. Reinforcement learning (RL) approaches have been applied to effectively modify the dynamic objects of the environment (e.g., pedestrians and other vehicles), also known as Non-Playable Characters (NPCs). However, existing RL approaches implement centralized controllers of the environment, resulting in possibly unrealistic and even invalid behaviors of the NPCs. In this paper, we propose to model NPCs as independent and fully autonomous agents to challenge the ADS under test (i.e., the ego ADS). In the first step of our approach, we train an adversarial ADS by designing a reward function as a linear combination of two components: (1) a component that encourages it to drive well in the given driving scenario, and (2) an adversarial component, that smoothly guides the agent towards a collision with the ego ADS. In our second step, we resume training of the ego ADS, to increase its robustness towards the behaviors of the adversarial ADS. Our experiments on a highway driving scenario show that the adversarial ADS is significantly more effective at generating collisions of the ego ADS than a random baseline. Moreover, adversarial retraining induces safe behaviors of the ego ADS, preventing the adversarial ADS from colliding.