Rapid Autonomy Transfer in Reinforcement Learning with a Single Pre- Trained Critic

Reinforcement learning (RL) is a well-studied framework to solve complex decision-making problems in unknown environments. The actor-critic model in RL facilitates autonomy transfer by allowing agents to iteratively update their policies using ongoing dynamic evaluations of value functions via a critic. In this paper, we examine the impact of using different pretrained critics on the performance of actor-critic algorithms. First, in any single given environment, we show that a pretrained critic can be effective in reducing the duration of an initial training phase, thereby accelerating convergence by a factor of up to 2×. In this setting, we identify the critical range of the number of episodes for which a critic will need to be trained in order for it to be an effective pretrained critic. Second, we show that a critic trained in one environment enables transfer of autonomy by aiding learning of behaviors in a different, yet related environment. We carry out extensive experiments on a bipedal locomotion task in the MuJoCo physics engine to verify our hypotheses. Our results in this paper mark the first step towards demonstrating the role and impact of pretrained critics to achieve rapid autonomy transfer for complex reinforcement learning tasks while minimizing costs associated with retraining in new environments.