Reinforcement learning of impedance control in stochastic force fields

Variable impedance control is essential for ensuring robust and safe physical interaction with the environment. As demonstrated in numerous force field experiments, humans combine two strategies to adapt their impedance to external perturbations: 1) if perturbations are unpredictable, subjects increase their impedance through co-contraction; 2) if perturbations are predictable, subjects learn a feed-forward command to counter the known perturbation. In this paper, we apply the force field paradigm to a simulated 7-DOF robot, by exerting stochastic forces on the robot's end-effector. The robot `subject' uses our model-free reinforcement learning algorithm PI2 to simultaneously learn the end-effector trajectories and variable impedance schedules. We demonstrate how the robot learns the same two-fold strategy to perturbation rejection as humans do, resulting in qualitatively similar behavior. Our results provide a biologically plausible approach to learning appropriate impedances purely from experience, without requiring a model of either body or environment dynamics.