RCDFNet: A 4-D Radar and Camera Dual-Level Fusion Network for 3-D Object Detection

The 4-D radar, as an advanced vehicle sensor, provides denser point clouds and elevation information than millimeter-wave (MMW) radar, establishing it as a valuable sensor for autonomous driving. Recently, the fusion of 4-D radar and camera has become a viable alternative to LiDAR for autonomous driving perception systems. However, existing fusion methods have not fully realized the potential of complementary advantages offered by these two modalities. To address this limitation, this study proposes RCDFNet, a dual-level fusion network for 3-D object detection with 4-D radar and camera. Specifically, we first exploit the geometric information from the 4-D radar to extract semantic image features from the perspective view (PV), yielding pseudo-camera features under the bird’s-eye view (BEV). In the first fusion level, the pseudo-camera features are integrated with the 4-D radar features using the spatial point fusion (SPF) module to obtain the radar-guided fusion features. In the second fusion level, we employ the deformable attention module (DA) mechanism to facilitate the interaction between camera-view-transform features and the radar-guided fusion features, to generate BEV-match fusion features. Furthermore, we propose the target-aware refinement (TAR) module to learn foreground targets’ occupancy adaptively in the BEV, mitigating the impact of the background noise of two sensors. We validate the RCDFNet on the TJ4DRadSet and View-of-Delft (VoD) datasets. The experimental results demonstrate that RCDFNet significantly outperforms the baseline RCFusion, achieving the state-of-the-art performance in 3-D object detection. Code and models are available at https://github.com/ D-Hourse/RCDFNet/tree/master