Sequential Decision MARL for Adaptive Traffic Signal Control With Different Intersections Priorities

Existing multi-agent reinforcement learning (MARL) in adaptive traffic signal control (ATSC) typically models cooperative control of multiple intersections as a cooperative Markov game, optimizing the average traffic efficiency of intersections with the same emphasis. However, it is insufficient to meet the requirements in real ATSC scenarios when all intersections are treated equally. To this end, this work proposes the Captain-Member Markov Game (CM-MG) that considers the different priorities between intersections. CM-MG categorizes intersections into special and ordinary intersections, controlled by captain agents and member agents to optimize the traffic efficiency of local and overall road networks, respectively. The cooperative requirements of CM-MG are achieved through a sequential decision-making principle. Captains have priority in choosing actions, and members make decisions sequentially, following the breadth-first traversal order in the road network after obtaining their precursors’ intentions. Then, a cooperative MARL algorithm, i.e., Sequential Decision Deep Graph Network (GNSD-Light), is proposed to learn the optimal joint policy that meets the learning goals of both captains and members. To be unrestricted by the scales of intersections, GNSD-Light adopts an autoregressive framework where all agents make decisions sequentially in a predetermined order by sharing the same decision model. In addition, to obtain sufficient state representation, two relative position encoding-based spatiotemporal representation modules are designed for GNSD-Light based on the characteristics of ATSC scenarios. Finally, through adequate experiments and qualitative analysis, we have confirmed that our method effectively balances traffic efficiency among both the overall road network and special intersections.