Offline Reinforcement Learning with Generative Adversarial Networks and Uncertainty Estimation

In recent years, offline reinforcement learning has attracted considerable attention in artificial intelligence. By generating a static dataset through a behavior policy, it is unable to engage in online interactions with the environment. However, this inevitably leads to states or actions undergoing inherent distribution shifts. To address the above concerns, offline reinforcement learning with generative adversarial networks and uncertainty estimation (GANUE) is proposed. This approach parameterizes the conditional generative adversarial networks and avoids the over-constraint problem that would be introduced when using a distance metric. Estimating the Q-value through ensemble uncertainty not only relaxes the policy constraint strength but also enhances the out-of-distribution generalization ability. The experimental results show that GANUE significantly performs better on the Maze2D and Adroit tasks than multiple baselines. In addition, we perform sensitivity analysis experiments for the parameters of the perturbation model.