True Online TD(λ)-Replay An Efficient Model-free Planning with Full Replay

In this paper, we present a new reinforcement learning prediction method that extends the capabilities of the true online TD(λ) to allow an agent to efficiently replay all of its past experience, online in the sequence that they appear with. We demonstrate that, for problems that benefit from experience replay, our new method outperforms true online TD(λ), albeit quadratic in complexity due to its replay capabilities. In addition, we demonstrate that our method outperforms other methods with similar quadratic complexity such as Dyna Planning and TD(0)-Replay algorithms. We showcase the capabilities of our method on two benchmarking domains, a random walk problem tested with simple binary features and on a myoelectric domain that is tested with features that are deeply extracted from sEMG signals. Experimental results confirm the particular suitability of this method for a deep architecture over other methods.