An Optimistic Approach to the Temporal Difference Error in Off-Policy Actor-Critic Algorithms

The balance between exploration and exploitation is a protracted challenge in reinforcement learning (RL). High dimensional continuous action domains increase the time and resources required to find an optimal policy due to the intractable actions, i.e., infinitely many possible. In this paper, we transfer the existing theories on animal motivational systems to the deep RL paradigm to constitute an efficient, directed exploration strategy for off-policy actor-critic algorithms. We introduce a novel and scalable high-frequency perturbation framework that directs agents to state-action pairs with large value errors through an additive deep explorer network. By maximizing the error induced by the Q-value estimation, our method forces the approximated critic function to learn the values of less-known or unselected actions and prevents premature behavior before sufficiently learning the environment. Moreover, the presented algorithm can benefit from the off-policy setting by extracting useful information from the experiences contained in the experience replay buffer, in contrast to the state-of-the-art exploration techniques, which exhibit poor performance in the off-policy setting due to the separated policy and exploration strategy updates. An extensive set of empirical studies demonstrates that our method considerably diversifies the visited states, outperforms the baselines, and improves the state-of-the-art off-policy actorcritic by a significant margin.