Preference-Based Reinforcement Learning for Autonomous Vehicle Control Considering the Benefits of Following Vehicles

Most studies developing car-following controllers for AVs in mixed traffic primarily focus on maximizing the utility of the AVs. However, the utility of the entire mixed traffic flow is largely ignored, which may lead to self-centered AVs. This study aims to develop a leading cruise controller for AVs that can “look behind” the mixed traffic flow. It enables the AV to adapt its car-following behaviors according to the states of both the leading human-driven vehicle (LHDV) and the following human-driven vehicle (FHDV). Car-following trajectories are extracted, processed, and analyzed based on the real-world autonomous driving dataset – the Waymo Open Dataset. Then car-following behaviors of HDVs are approximated through an inverse reinforcement learning (IRL) approach. After that, this study proposes a preference-based soft actor-critic (PbSAC) algorithm to optimize the speed of AVs in the three-vehicle car-following scenario. In addition to safety, efficiency, and string stability for AV, these metrics for FHDV are also included in the reward function. An adaptive preference-adjusting module is developed to update the weights in the reward function based on expert knowledge. Experimental results indicate that the proposed algorithm can improve safety, efficiency, and string stability for both AV and FHDV. Moreover, if behavioral changes of human drivers when they are following AVs are not accurately captured, the learned AV controller may result in suboptimal performance.