Dynamic Arterial Coordinated Control Based on Multi-agent Reinforcement Learning

In order to effectively reduce the average travel time of vehicles on urban traffic arterial roads, multi-agent reinforcement learning (MARL) is introduced into the arterial coordinated control. Multi-agent RL (MARL) overcomes the scalability issue by distributing the global control to each local RL agent, but it introduces new challenges: now the environment becomes partially observable from the viewpoint of each local agent due to limited communication among agents. Inspired by independent Q-learning, we proposed multi-agent asynchronous advantage-critic (MA2C) algorithm for arterial traffic signals control (ATSC). Moreover, we use the decentralized parameter sharing training protocol to mitigates the slowing of convergence due to nonstationarity. In order to improve the safety of MA2C arterial road coordination control on the actual road and ensure the flexibility of coordination, a new action definition is proposed in combination with NEMA phase. We compares the performance of agents using different reward functions in a simulation of a synthetic arterial road with 5 intersections. The experimental results show that the performance of the MA2C algorithm with different reward functions all outperform the traditional green wave method, in which minimizing waiting time results in the lowest average travel time. Another set of experiments shows that our action definition based on NEMA phase is more effective than the traditional “four stage” scheme in arterial traffic signals control.