TEMPO: Timestep Explanations for Modeling Preferences in Online Preference-Based RL

Human-in-the-loop reinforcement learning enables non-expert users to train agents through interactive feedback. While preference-based methods (PbRL) have gained popularity by allowing humans to express preferences between trajectory pairs, they are limited by their narrow feedback interface - humans can only indicate which trajectory they prefer but not why. This work introduces a more expressive preference-based learning framework that allows humans to provide step-level explanations alongside their preferences, indicating which specific timesteps influenced their decisions. Our key technical innovation is a training approach that incorporates these explanations using a novel loss function combining reward learning, saliency-based explanations, and structural regularization. We evaluate our method across multiple continuous control environments using both synthetic oracles and human trainers. Results show that our approach accelerates learning compared to standard preference-based methods while maintaining robustness across different types of human feedback patterns. The improved learning speed and more natural interaction suggest that this is a promising direction for making human-in-the-loop RL more practical.