Learning From Demonstrations: A Computationally Efficient Inverse Reinforcement Learning Approach With Simplified Implementation

Reinforcement learning (RL) research usually requires a reward function from sophisticated domain knowledge to perform well. Inverse reinforcement learning (IRL) methods provide the opportunity to recover such reward functions from existing expert demonstrations. However, current IRL methods require intensive computation and excessive memory storage when the state space increases. To this end, we propose a computationally efficient inverse reinforcement learning (e-IRL) approach to 1) simplify the gradient algorithm for the reward network; 2) implement the loss function with feature expectation instead of state visiting frequency; and 3) enable fast, accurate, and automatic feature processing. Specifically, we design a new featurization network to accommodate trajectories and automatically output the aligned feature vectors without human intervention. In addition, this proposed approach derives simplified algorithm formulas to obtain a gradient of loss function to update weights for the reward network. This approach eliminates repeated calculation and excessive storage of gradient parameters from demonstrated trajectories in literature. The feature expectation computation for both the expert (one-shot) and learner (streamlined) is employed to efficiently obtain the important gradient parameters. Three examples are considered to validate the effectiveness of the proposed method, which shows better performance in terms of average return, required average number of steps, and required number of demonstrations than comparable methods.