Integrating Vision and Language Foundation Models for Enhanced Navigation and Decision-Making in Connected Autonomous Vehicles

Traditional connected autonomous vehicles (CAVs) face challenges in personalized navigation and human-like driving decision-making. In this paper, we propose a self-driving system based on vision and language foundation models to address these issues. Specifically, we first design a ChatGPT-4-based vision-and-language navigation (VLN) model that navigates according to environmental observations and routes described in user instructions, demonstrating personalized navigation capabilities. The VLN model has a local navigation experience database that can be uploaded via wireless communication to roadside units (RSUs) for sharing with other vehicles, thereby enhancing ChatGPT's VLN capabilities. Next, we develop a ChatGPT-4-based knowledge-driven driving decision model that enables it to generate driving actions such as acceleration, deceleration, and lane-change based on knowledge, akin to human drivers. This driving decision model includes a local driving experience database that can be shared with other vehicles, further augmenting ChatGPT's driving decision capabilities. Finally, we design an obstacle-repulsion field-based lane-changing path planner for the lane-changing driving action, utilizing a combination of dynamic programming (DP) and quadratic programming (QP) to achieve a safe and smooth lane-changing path. Experimental results demonstrate that the system's VLN model, driving decision model, and lane-changing path planner respectively improve task completion (TC) by 6.1%, success rate (SR) by 5.4%, and curve smoothness by 21.4% compared to the baseline, underscoring the superiority of the system.