Balanced Prioritized Experience Replay

As an important technique of deep reinforcement learning, prioritized experience replay (PER) helps agents to learn and update strategies through past experience. However, PER does not change the situation that the historical information learned by the agent and stored in the replay buffer is fixed and unchanging. On the contrary, to learn a good strategy efficiently, agents often require the latest interactive information. In this paper, we propose a new method, Balance Prioritized Experience Replay (BPER), for properly prioritizing experience so as to replay recent important transitions more frequently, and therefore learn more efficiently. We incorporate BPER into Twin Delayed Deep Deterministic policy gradient algorithm (TD3) to generate a new policy gradient algorithm called TD3-BPER. The experimental results show that TD3-BPER can achieve better performance than original TD3 in several environments, such as HalfCheetah, Walker2d, Ant, and Swimmer.