Towards Cooperative Driving among Heterogeneous CAVs: A Safe Multi-Agent Reinforcement Learning Approach

With the advancement of Intelligent Transportation Systems and Vehicle-to-Everything communication technologies, the future traffic scenario is anticipated to be a mixed environment consisting of heterogeneous Connected and Autonomous Vehicles (CAVs) and Human-Driven Vehicles. Though reinforcement learning (RL) has shown tremendous potential in solving decision-making problems in autonomous driving, existing research often overlooks the heterogeneity of CAVs. Moreover, current RL methods lack robust safety guarantees, especially in mixed traffic scenarios where HDVs are often non-cooperative and uncontrollable. To address challenges of the cooperative driving among heterogeneous CAVs in dynamic and mixed traffic, we have developed a novel heterogeneous cluster safe algorithm and hybrid-critic training framework. Specifically, to overcome the limitations of existing methods primarily designed for homogeneous agents, we introduce a cluster policy-sharing mechanism. Additionally, we propose a safe policy update method with a high sample utilization rate, which combined with the cluster policy-sharing mechanism leads to the HCP4O algorithm. Furthermore, we have designed an innovative hybrid-critic training framework to enhance the safety driving performance of heterogeneous CAVs under various conditions. Finally, we train our model in hundreds of random and dynamic scenarios to validate the generalization ability of agents. The experimental results show that, compared to current advanced Multi-Agent RL algorithms and rule-based methods, our approach demonstrates significant advantages in safety and efficiency, especially in scenarios with a large number of agents.