Adaptive Relay Hindsight Experience Replay for Sequential Manipulation Tasks with Sparse Rewards

In reinforcement learning, the sequential manipulation tasks with sparse rewards remain a great challenge. Recent approaches such as relay hindsight experience replay (RHER) expedite learning by reusing the already-learned policy with the self-guided exploration strategy (SGES). However, SGES causes decision conflicts between different policies, compressing the exploration time of the next stage, which in turn makes the number of samples in the new stage decreases. To address this problem, we propose a novel approach called Adaptive Relay Hindsight Experience Replay (Adaptive-RHER). The core of Adaptive-RHER is Adaptive Self-guided Exploration Strategy (A-SGES), which can adaptively adjusts the sampling probability of different policies based on the expected time to achieve each goal and the decision-making performance of the complex policy. Extensive experiments have been conducted with the proposed Adaptive-RHER in different sequential manipulation tasks and experimental results show that Adaptive-RHER outperforms both model-free and model-based baselines.