TD(0)-Replay: An Efficient Model-Free Planning with full Replay

In this paper we present a novel reinforcement learning method that allows for full replay of all past experience in every step of a reinforcement learning agent life with reasonable overhead. In particular, we show how to deduce an efficient equivalent backward view by replaying the full past experience online using TD(0) error for a linear model. We call the resultant methods TD(0)-Replay and Sarsa(0)-Replan, respectively. We emphasise the already established link between replaying and planning in our algorithm design by comparing it with an Extensive Dyna Planning algorithm, where we show that our method can outperform this expensive form of planning methods. We test the new methods on two different domain problems; Random Walk to test TD(0)-Replay prediction capabilities and Dyna Maze to test Sarsa(0)-Replan planning capabilities where we show that our algorithms dominate other replaying and planning methods.