Reward Redistribution for Reinforcement Learning of Dynamic Nonprehensile Manipulation

Recent reinforcement learning (RL) systems can solve a wide variety of manipulation tasks even in real-world robotic implementations. However, in some nonprehensile manipulation tasks (e.g. poking, throwing), the classical reward system fails as the robot has to manipulate objects whose motion trajectory is partly uncontrollable. Such tasks require a specific type of reward that would reflect this temporal misalignment. We propose a novel method, based on a delayed reward redistribution, that allows a robot to fulfill goals in an only partially controllable environment. The reward system in our architecture combines information from other sensors together with inputs from an unsupervised vision module based on a variational autoencoder (VAE). This delayed reward system then controls the training of the motor module based on a Soft Actor-Critic (SAC) neural network. We compare results for a delayed and nondelayed version of our system in a simulated environment and show that the delayed reward greatly outperforms the nondelayed version.