RC-ROSNet: Fusing 3D Radar Range-Angle Heat Maps and Camera Images for Radar Object Segmentation

Modern millimeter-wave (mmWave) radar heat map-based object detection techniques have shown significant potential. However, radar heat maps still pose challenges for fine-grained object differentiation, and fusing camera maps with radar heat maps remains challenging. This paper proposes a multi-sensor radar object segmentation (ROS) method that fuses radar range-angle (RA) heat maps with camera RGB maps, leveraging the advantages of camera sensors for extracting semantic information and mmWave radar for extracting object position information. A method for mapping RGB to the RA coordinate system (RMTR) is proposed, eliminating the need for additional training branches and sensor calibration. A Transformer is utilized to predict the range and angle matrices of each object separately from a global perspective. These matrices then undergo an element-wise multiplication (Hadamard product) operation to generate the final pseudo-RA map. For the fusion method, this paper introduces the cross-attention fusion module (CAF), which uses the cross-attention mechanism to achieve more efficient fusion by treating the pseudo-RA feature map as the Query and the RA feature map as the Key and Value. Extensive experiments show that the proposed method achieves state-of-the-art (SOTA) performance on the CRUW and CARRADA datasets, with fewer parameters and reduced computational complexity. Our code for RC-ROSNet is available at https://github.com/Zhuanglong2/RCROSNet.