Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration

Deep reinforcement learning faces challenges in long-horizon, high-dimensional robotic tasks that provide rewards only upon completion. In this context, while expert demonstrations can help, they are costly. In this work, we leverage a sequential decomposition bias to learn control policies for such tasks from a single demonstration. Our method learns a goal-conditioned policy that guides the system through lowdimensional intermediate goals, offering more flexibility than using states as goals. However, this approach can complicate the task of ensuring compatibility between successive goals. To address these challenges, we propose a formal framework, GCSeqMDPs, and an algorithm, STIL, which properly handles a hindsight goal relabeling mechanism. We first demonstrate the benefits of STIL on a relatively simple, long-horizon, nonholonomic robotic task. Then, we show that STIL achieves unprecedented sample efficiency on more complex simulated tasks, such as humanoid locomotion and fast running on the Cassie robot, marking a significant step toward the resolution of complex robotic tasks with minimal task-specific information.