Improving Deep Reinforcement Learning with Intrinsic Rewards via Self-Imitation Learning

Deep reinforcement learning (RL) often employs intrinsic rewards to promote effective exploration in sparse reward environments. However, the coefficients for these intrinsic rewards are typically set to a uniform value throughout the learning process. This practice negatively impacts the performance of the deep RL agent by limiting its adaptability to environmental complexities, thereby affecting the balance between exploration and exploitation. Self-imitation learning (SIL) can mitigate this negative impact by selectively replicating successful past experiences. However, it also disrupts the learning of the deep RL agent using intrinsic rewards, thereby affecting the agent's performance. To prevent SIL from disrupting the learning of the deep RL agent employing intrinsic rewards and to enhance the performance of the agent, we propose Intrinsic Rewards-Based Self-Imitation Learning (IRSIL) for the on-policy actor-critic deep RL method that utilizes intrinsic rewards. Unlike traditional SIL, IRSIL allows the actor-critic deep RL agent employing intrinsic rewards to only imitate past experiences that facilitate the agent in learning optimal behaviors without disrupting the agent's learning. Specifically, IRSIL stores episodes yielding positive extrinsic rewards into the experience replay buffer. It then makes the actor-critic deep RL agent imitate state-action pairs from this replay buffer exclusively when their extrinsic returns exceed the agent's current value estimates, which are estimated based on both extrinsic and intrinsic rewards. These experimental results in sparse reward procedurally generated MiniGrid environments demonstrate our approach IRSIL significantly enhances the learning and performance of the actor-critic deep RL agent utilizing intrinsic rewards. Furthermore, IRSIL surpasses the performance of both traditional and state-of-the-art SIL approaches in these settings.