Enhancing Negotiation Policies via Spatio-Temporal Directed Graphs for Autonomous Interaction

Mixed traffic environments, comprising autonomous vehicles (AVs) and human-driven vehicles (HDVs), present substantial challenges for developing negotiation policies. These policies are essential for enabling AVs to make adaptive decisions and achieve harmonious interactions with HDVs in dynamic and complex scenarios. Despite their potential in autonomous driving, reinforcement learning-based decision-making schemes remain insufficient in addressing interaction-critical situations. To overcome this limitation, this study proposes a graph-enhanced negotiation-aware policy optimization (GNPO) framework, which embeds interaction awareness into all key components, including representation, understanding, and response. To support interaction representation, the spatio-temporal directed graph (STDG) captures the dynamic and asymmetric nature of interactions. It integrates multiple directed graph topologies, each encoding distinct social priors, to collectively depict realistic interactive behaviors. Additionally, a multimodal feature extractor fuses environmental perception, interaction cues, and task-related information, enabling comprehensive state understanding. Building on this representation, GNPO incorporates an interaction-critical reward function and an entropy-regularized adaptive policy optimization scheme, both designed to promote harmonious and context-aware behaviors in dense traffic scenarios. The proposed framework is validated in both interaction-intensive simulated environments and real-data-based digital twin scenarios, with its effectiveness quantitatively demonstrated by substantial improvements in success rate, safety, and efficiency. Qualitative evaluations further show that GNPO is capable of generating human-like cooperative and competitive behaviors. Note to Practitioners—Practitioners aiming to deploy decision-making algorithms in mixed traffic should pay special attention to modeling asymmetric and dynamic interactions. It is important to ensure real-time processing capability when applying graph-based methods such as GNPO. High-quality traffic data, reliable interaction cues, and efficient graph computation frameworks are key to effective implementation. To facilitate deployment in diverse environments, robustness is improved by including rare interaction cases and adversarial behaviors in training.