Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMs

Large Language Models (LLMs) have demonstrated remarkable abilities in various language tasks, making them promising candidates for decision-making in robotics. Inspired by Hierarchical Reinforcement Learning (HRL), we propose Retrieval-Augmented Hierarchical in-context reinforcement Learning (RAHL), a novel framework that decomposes complex tasks into sub-tasks using an LLM-based high-level policy, in which a complex task is decomposed into sub-tasks by a high-level policy on-the-fly. The sub-tasks, defined by goals, are assigned to the low-level policy to complete. To improve the agent's performance in multi-episode execution, we propose Hindsight Modular Reflection (HMR), where, instead of reflecting on the full trajectory, we let the agent reflect on shorter sub-trajectories using intermediate goals to improve reflection efficiency. We evaluated the decision-making ability of the proposed RAHL in three benchmark environments, ALFWorld, Webshop, and HotpotQA, where the results show that RAHL can achieve an improvement in the performance of, respectively, 9%, 42%, and 10% in 5 execution episodes compared to state-of-the-art baselines. We also implemented RAHL on the Boston Dynamics SPOT robot, which is shown to effectively scan the environment, find entrances, and navigate to new rooms controlled by the LLM policy.