Adversarial Learning of Failure Prevention Policies

In order to ensure safety in robot manipulation, it is essential for robots to experience potential failures that may give rise to unsafe situations. Experiencing numerous potentially unsafe situations in the real world can be both unsafe and inefficient. Hence, experiencing these unsafe situations in a simulated environment is much more convenient. In this work, we propose an adversarial reinforcement learning system that includes adversary components to learn safe manipulations. Adversarial policies that generate failures are learned using reinforcement learning. Then, these adversarial policies are used to increase the risks of failures and to learn a robust protagonist policy that reduces the risk of failures in an adversarial reinforcement learning framework. The proposed system is tested on a humanoid robot that stirs the contents of a bowl. Failures of overturning the bowl, sliding the bowl, and spilling contents from the bowl that occur in this scenario are produced and a protagonist policy that prevents these failures is learned. The results demonstrate that learning the protagonist policy with the proposed system enhances the safety of robot manipulation effectively.