Interaction-Aware Hierarchical Representation of Multi-Vehicle Reinforcement Learning for Cooperative Control in Dense Mixed Traffic

Improving traffic performance in dense mixed traffic scenarios such as bottlenecks, presents significant challenges due to complex interactions and the unpredictable behaviors of human drivers. These challenges are compounded by varying human driving styles and different proportions of Connected and Automated Vehicles (CAVs) within the traffic flow. Our research focuses on developing cooperative control strategies for CAVs to enhance generalization across diverse traffic scenarios. To address these challenges, we introduce an Interaction-Aware Hierarchical Representation (IAHR) module, integrated into Multi-Agent Reinforcement Learning (MARL) framework. The IAHR module hierarchically processes interactions between CAVs and Human-Driven Vehicles (HDVs), effectively extracting essential features to facilitate generalization across various traffic scenarios. Additionally, we design an effective reward function that balances individual interests with overall traffic performance, guiding CAVs to improve their driving efficiency and safety while also enhancing overall traffic flow. The model is rigorously trained and zero-shot evaluated in various bottleneck scenarios. Results demonstrate the model's capability to significantly improve traffic performance under dense conditions and generalize across different CAV penetration rates, vehicle numbers, and HDV driving style distributions.