Gamma-Regression-Based Inverse Reinforcement Learning From Suboptimal Demonstrations

Inverse reinforcement learning (IRL) is a technique that estimates the intention of an expert who acts optimally on a specific intention, as a reward from demonstration (i.e., recorded data of the expert’s behavior). Traditional IRL algorithms are based on the assumption that expert demonstrations are optimal. However, capturing perfectly optimal demonstrations is unrealistic because suboptimal demonstrations with nonoptimal behavior are common in practice. To address this limitation, we first model a suboptimal transition distribution as a contaminated distribution, wherein nonoptimal transitions are mixed as outliers into an optimal transition distribution. Then, using a logistic-regression-based IRL as an example, we propose a new method to replace the cross-entropy function with a gamma binary cross-entropy function, thereby establishing an objective function based on gamma divergence. The experimental results indicate that the proposed method is effective in estimating the reward for obtaining a near-optimal policy from suboptimal demonstrations.