Enhancing Cooperation of Vehicle Merging Control in Heavy Traffic Using Communication-Based Soft Actor-Critic Algorithm

A promising way to improve efficiency in highway on-ramp regions is to control connected and automated vehicles (CAVs) to pass the merging section sequentially. The primary objective of this paper is to incorporate the reinforcement learning (RL) technique into vehicle merging control to achieve global cooperation. A communication protocol among RL agents is integrated with the Soft Actor-Critic (CSAC) algorithm. The parallel SAC agents with a parameter-sharing structure cooperate to optimize the common reward function. It enables CAVs to know the actions of each other so that they proactively negotiate to adjust speeds. We designed a parsimonious state representation containing crucial merging information to speed up the RL training. The effects of the merging control strategy were investigated with the simulation model in scenarios with different CAV penetration rates and traffic flow compositions. Results showed that the CSAC-based merging strategy generated collision-free merging trajectories with a short travel time while guaranteeing traffic safety in various traffic conditions. Four baselines (including two rule-based and two RL-based merging strategies) were applied in the same scenarios for comparison. It was found that the proposed merging strategy took the safest merging behaviors (zero traffic conflicts) and reduced the most significant travel time (56.9%).