AVLLM-Based Multimodal Reasoning Segmentation and Detection Approach for Intelligent Driving

Reasoning segmentation and detection are important to realize secure and reliable intelligent driving. In this paper, we propose an Audio-Vision-Language Large Model-based Multimodal Reasoning Segmentation and Detection (AVLLM-MRSD) approach, where audio prompts or text prompts in Chinese or English are allowed when executing reasoning segmentation and detection tasks. The proposed AVLLM-MRSD approach can simultaneously output reasoning language response and segmented-detected image. Specifically, the multimodal audio-language large model (ALLM) module which is composed of the audio encoder and large language model (LLM) is first employed to convert the audio prompt to the text prompt. Then, the text prompt and image are input to the multimodal vision-language large model (VLLM) module to obtain the reasoning language response and segmented-detected image. Besides, the proposed method supports multi-round and multi-object reasoning segmentation and detection tasks. Extensive experimental results in real traffic scenarios show that the proposed AVLLM-MRSD approach can more efficiently and accurately complete reasoning segmentation and detection of various vehicles, pedestrians, lanes, barriers, and traffic lights compared with the existing LISA and Grounded SAM methods.