Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles

The future of autonomous vehicles lies in the convergence of human-centric design and advanced AI capabilities. Autonomous vehicles of the future will not only transport passengers but also interact and adapt to their desires, making the journey comfortable, efficient, and pleasant. In this paper, we present a novel framework that leverages Large Language Models (LLMs) to enhance autonomous vehicles' decision-making processes. By integrating LLMs' natural language capabilities and contextual understanding, specialized tools usage, synergizing reasoning, and acting with various modules on autonomous vehicles, this framework aims to seamlessly integrate the advanced language and reasoning capabilities of LLMs into autonomous vehicles. The proposed framework holds the potential to revolutionize the way autonomous vehicles operate, offering personalized assistance, continuous learning, and transparent decision-making, ultimately contributing to safer and more efficient autonomous driving technologies.

1. Introduction

Recently, Large Language Models (LLMs) have attracted significant attention. The key to their success lies in their remarkable ability to process a wide range of word-based inputs, including prompts, questions, dialogues, and vocabulary spanning diverse domains, resulting in significant and coherent textual outputs. LLMs serve as vast store-houses of abundant information and knowledge acquired from numerous texts, much like the human brain. Considering the LLM's ability to emulate the human brain functions, it prompts us to ask: could we leverage the impressive capabilities of LLMs to revolutionize the future of autonomous driving?