Mixed Time-Frame Training for Reinforcement Learning

Reinforcement learning typically only uses one type of environment during training: episodic or non-episodic. In this paper, we propose a novel training technique, Mixed Time-Frame Training, which combines episodic environments and non-episodic environments through training in parallel. This approach takes advantage of the different exploration-exploitation characteristics of the two environment types. We shall benchmark the proposed technique on 10 Atari games with varying levels of exploration difficulty, showing a significant increase in performance. The increase in performance comes with no increase in training frames, algorithmic complexity or computation.