Improving Radar–Camera Fusion for 3D Object Detection via Eliciting Knowledge From Foundation Model

Fusing camera and radar signals to establish reliable perception systems for autonomous driving has been a prevalent solution due to its low deployment costs. Existing approaches primarily investigate cross-modal interaction mechanisms to facilitate effective radar information exploitation. Despite progress, current camera-radar fusion methods for 3-D object detection still face challenges in overcoming sparsity and noise. To this end, by eliciting knowledge from foundation model, we propose a novel foundation model enhanced radar–camera fusion (FRCFusion) model for 3-D object detection that consists of two main components: 1) Progressive Radar Signal Densifying (PRSD), which regards sparse radar depths as guiding prompts to elicit scene comprehension knowledge from foundation models, and 2) multiscale cross-modal detection (MCD), which enables thorough fusion of multiscale features. For PRSD, we prompt foundation model in an iterative manner, resulting in reliable densified radar signals. For MCD, we retain multiscale semantic information from each modality, enabling dynamic alignment, and comprehensive interaction between different scales of radar and camera features. Extensive experimental results demonstrate state-of-the-art performance on benchmarks with significant improvements in detection accuracy and robustness.