A LLM-based Multimodal Warning System for Driver Assistance

Advanced driver assistance systems (ADAS) play a crucial role in enhancing road safety by providing timely warnings and assistance. However, drivers exhibit varying hazard perception abilities and interaction needs due to their distinct characteristics, necessitating personalized warning systems to improve user experience and acceptance. Despite most advanced systems today are able to offer personalization, they remain static based on user setting rather than updating automatically. Large Language Models (LLMs), known for their exceptional knowledge acquisition, reasoning, and human-machine interaction skills, offer a promising solution for developing more customized systems. In light of this, we developed a LLM-based Multimodal Warning system (LLM-MW), which offers personalized warnings through multimodal interaction. By interpreting the traffic environment and drivers' profiles, the system can plan multimodal warning contents and perform warnings. Our experiment results show that the LLM-MW provides a level of personalization for different drivers in various scenarios.