Sample-Efficient Imitation Learning via Reconstructed Intrinsic Reward

Sample inefficiency has remained a main challenge in existing imitation learning methods. We attribute this inefficiency to the use of random exploration strategies, which collect many repeated state samples that are dispensable in training the agent. In this paper, we propose a method termed reconstructed intrinsic reward (RIR), which encourages the agent to bypass explorations of repeated states by adding an exploration bonus to the reward function. RIR serves as an incentive mechanism that rewards the agent with a high bonus if it collects a sample rarely encountered in previous explorations; otherwise, it gives a low bonus if a state is repeatedly gathered many times. The bonus is computed as the relative quaternion error of input samples by solving a reconstruction problem of collected states. Experiments show that our method outperforms state-of-the-art imitation learning methods in sample efficiency. Ablation studies prove that this efficiency comes from a more advisable exploration strategy used by RIR.