Efficient Hierarchical Exploration with An Active Subgoal Generation Strategy

Goal-conditioned hierarchical reinforcement learning (HRL) is a promising approach in long-horizon and complex tasks. It has been successfully used by decomposing tasks via subgoals. However, the existing goal-conditioned HRL methods often suffer from training inefficiency in the long-horizon tasks with sparse external rewards. For this problem, this paper proposes a novel framework that can train high-level policy efficiently with an effective sub goal generation method. The key component of the framework is an active subgoal generation strategy which consists of (a) an efficient high-level policy learning mechanism and (b) two measures of novelty and coverage for filtering effective subgoals. We also introduce the HER mechanism to the framework to solve the problem of sparse reward in the environment. Building upon the proposed framework, this paper develops an active hierarchical exploration strategy with an efficient subgoal generation method. We test our method on a variety of continuous control tasks. Experimental results demonstrate that the strategy outperforms the state-of-the-art HRL approaches.