An Efficient Synchronous Training Integrated Model for Driving Decision-Making Based on Deep Reinforcement Learning

Deep reinforcement learning (DRL) is a promising way to develop autonomous driving decision-making models. However, poor driving decisions and low sample efficiency for multiple DRL coupled training hinder its applications in driving decision-making models. This article proposes an innovative framework to combine two different DRL algorithms as the upper- and lower-layer planner to make car-following and lane-changing decisions respectively. The upper- and lower-layer models are trained simultaneously, and the double-layer model outputs a composite driving action. In the upper-layer model, using TD3 algorithm generates continuous vehicle speed. This article proposes the action exploration mechanism where the TD3 algorithm selects one of two action policies with a probability and then outputs an action value in the early training phase. Moreover, the proposed Q-value auxiliary networks guide our upper-layer algorithm to compute the Q-value based on a Q-value from the trained TD3 algorithm. Dueling-double DQN is used to address the issue of how a vehicle changes lanes in the lower-layer model and to output discrete values to instruct the vehicle to change lanes. To validate our model in autonomous driving applications, different training and testing scenarios simulating expressways are designed through SUMO. The experiments show that our methods address the difficulty of coupling training for the integrated model and improve its performance under different traffic flow scenarios. Compared with other models, our model enhances driving velocity while ensuring vehicle safety.