Heuristic Reward Function for Reinforcement Learning Based Manipulator Motion Planning

In recent years, the motion planning algorithm based on reinforcement learning has shown its potential to meet the requirements of high-dimensional planning space, complex task environments, and fast online planning simultaneously. However, the performance of the reinforcement learning-based motion planning algorithm for manipulators is closely related to the quality of the reward function, which is typically designed manually. In this paper, inspired by heuristic functions, a new learning-based reward function design framework is proposed. By relaxing the constraint of the original motion planning problem, a simple task is established. The cost of a solution to the simple task is taken as a heuristic for the original problem, and the heuristic is then incorporated into the reward function. We tested the performance of heuristic-guided reward functions based on the soft-actor-critic algorithm in various difficult scenarios. The experimental results showed that our reward function design method improves the convergence speed of the reinforcement learning training process and the planning success rate of the policy obtained after the training. In addition, the trained motion planning policy could be directly transferred to a real manipulator for online execution without modification, thus verifying the flexibility of the strategy between simulated and real environments.