Air-to-ground shepherd problem: An action-delay reinforcement learning approach

In this paper, we consider an optimization problem motivated by the International Aerial Robotics Competition (IARC) Mission-7, or the shepherd action. IARC Mission-7 requires an autonomous drone (i.e. the shepherd dog) to drive ground vehicle (sheep) across the green-line boundary of an competition arena of 20m × 20m within 10 mins. There are two actions, either top touch or collision to change the motion of ground vehicle(GV). The policy of the drone is to choose target and action type, as decision-making. Within a action process, the aerial robot takes certain period of time to fly from current position to the target. This period of time is named as action delay in this work, which is a key feature for reward generation and decision-making. In order to predict action delay before actual implementation, the path integral control method is applied to generate trajectories to the target in the environment with moving obstacles. The pre-trained values of every action on each state are stored in a discrete multi-dimensional tabular system, and are extracted for decision-making module. The simulation results in the IARC Mission-7 scenario validate that our methods proposed in this paper are effective.