Sample efficient transfer in reinforcement learning for high variable cost environments with an inaccurate source reward model

Here we propose an algorithm that combines two classic ideas, transfer learning and temporal abstraction, to accelerate learning in high variable cost environments (HVC-envs). In an HVC-env, each sampling of the environment incurs a high cost, thus methods to accelerate learning are sought to reduce the incurred cost. Transfer learning can be useful for such environments by using prior knowledge from a source environment. As only a small number of samples can be collected from an HVC-env due to high sampling cost, learning becomes challenging when the source environment provides inaccurate rewards. To overcome this challenge we propose a simple but effective way of creating useful temporally extended actions from an inaccurate physics guided model (PGM) that acts as the source task. At first we address this issue theoretically by providing performance bounds between two semi-Markov Decision Processes (SMDPs) with different reward functions. Later we develop two benchmark HVC-envs where learning must happen using a small number of real samples (often on the order of ~ 10² or 10³). Finally we show that it is possible to obtain sequential high rewards in both of these environments using ~ 10³ real samples by leveraging knowledge from PGMs with inaccurate reward models.