Efficient Policy Learning Based on Pretrained Vision-Language Model Representations

Reinforcement learning is a powerful framework for training intelligent agents to make optimal decisions in various environments. Several recent studies have analyzed and discussed the efficacy of utilizing pretrained language model features within the reinforcement learning paradigm. In this paper, we examine and investigate the impact of employing pretrained vision-language models, specifically the contrastive language image pretraining (CLIP) approach, in the policy gradient method. It is found that the intrinsic rewards will have a negative impact on learning with pretrained model features in some complex environments. To address this issue, we introduce a self-imitation guided policy gradient (SGPG) method to avoid using intrinsic rewards while enhancing learning efficiency and performance. This approach involves self-imitation learning to train a guiding policy, employs policy gradient techniques to train the target policy, and utilizes a KL-divergence to ensure the action distribution of the target policy closely approximates that of the guiding policy. The rationale behind learning two distinct policies is to prevent the simultaneous optimization of disparate objectives within a single policy learning process, which could result in suboptimal performance. In our experiments, we assess the proposed model using several sparse reward procedurally generated tasks and partial observations as inputs. The results show the effectiveness of incorporating pretrained model features in the policy gradient approach, and empirically validate the superior performance of our proposed method compared to baseline techniques and some recent methods. Some limitations of using pretrained model feature are also found in high-dimensional environments, which is discussed and points out the further work.