Accelerated Inverse Reinforcement Learning with Randomly Pre-sampled Policies for Autonomous Driving Reward Design

To learn a reward function that a driver adheres to is of importance to the human-like design of autonomous driving systems. Inverse reinforcement learning (IRL) is one of the recent advances that can achieve this objective, but it often suffers from the low efficiency of generating optimal policy by reinforcement learning (RL) each time when updating reward weights. This paper presents an accelerated IRL method by approaching the optimal policy among randomly pre-sampled policies in designed sub-space instead of finding it through RL in the whole policy space. The corresponding trajectories are targeted via an optimal trajectory selector in the candidate trajectory library generated by pre-sampled policies. The weights then are updated by comparing the selected trajectories and the expert ones. The proposed method is very suitable for improving learning efficiency for low-dimensional problems like autonomous driving, whose expert policies are nearly tractable. Results with simulated driving data show that it only took 11 iterations to converge, while the average longitudinal RMS error of the recovered trajectories based on the learned reward function was only 2.14m.