Improving the Learning Efficiency of Hierarchical Reinforcement Learning SUHRL Using Transition Information Between Subgoals

The hierarchical reinforcement learning approach incorporates intrinsic motivation into reinforcement learning. In this approach, the agents are hierarchically organized into a higher-level strategy that learns the application order (plan) of sub goals and a lower-level strategy that learns the sequence of actions up to the subgoals. This incorporation has been shown to facilitate problem-solving even in environments with sparse rewards. Stepwise Unified Hierarchical Reinforcement Learning (SUHRL) has been proposed as a hierarchical reinforcement learning algorithm. SUHRL enables autonomous learning without the need for prior knowledge or predefined subgoals. However, learning efficiency is low, and, in some cases, problem solving is not possible. Therefore, we propose the enhancement of SUHRL by constraining the exploration of sub goals based on the number of steps required to transition between them. This improvement is based on Hierarchical Reinforcement Learning with a k-step Adjacency Constraint, a method that can achieve efficient learning when sub goals are predefined. Experimental results using the Frozen Lake environment show that the proposed method exhibits a better learning efficiency than SUHRL.