Stability Analysis in Mixed-Autonomous Traffic With Deep Reinforcement Learning

The emergence of autonomous driving vehicles on roads has increased the importance of research on autonomous driving in mixed-autonomous traffic. In mixed-autonomous traffic scenarios, it is necessary to comprehend the instability of autonomous vehicles and traffic flow corresponding to the uncertainty level in human-driven behaviors. However, studies of stability analysis in deep reinforcement learning are limited. This study focuses on the impact of deep reinforcement learning based autonomous vehicles in mixed-autonomous traffic from the stability perspective. We define the policy instability and traffic flow instabilities using the entropy of the velocity distributions to quantitatively measure the instability of an autonomous vehicle. Subsequently, we provide mathematical analyses to explain logarithmic growth patterns of instability. Moreover, we propose a novel deep reinforcement learning approach that jointly determines discrete and continuous actions under partial observation. To verify the proposed solution, we perform extensive simulations of various traffic scenarios (e.g., increasing traffic volumes, increasing the number of autonomous vehicles on the road, and setting the multiple uncertainty levels for human-driven behaviors) with ablation studies on reward function. Moreover, we analyze instabilities when human-driven vehicles are modeled using the human-like noisy controller and a policy that imitates actual human-driving data based on imitation learning. The simulation results support the theoretical analysis and confirm that the proposed method is stabler compared to a conventional control-theoretic approach.

I. Introduction

Autonomous driving is a rapidly evolving topic and is attracting significant attention from both academia and industry. As autonomous driving vehicles begin arriving on the roads, studies on mixed-autonomous traffic that review the scenario of autonomous vehicles and human-driven vehicles cohabitating on the road are also being highlighted. Particularly, studies on the impact of autonomous vehicles on human drivers are being focused on. The power of autonomous vehicles in heavy traffic flow control was first shown in [1], which experimentally demonstrated that a single autonomous vehicle successfully reduces stop-and-go waves [2] of more than 20 human-driven vehicles on a ring road.