LiDAR-Camera cross fusion network towards 3D object detection in self-driving

In recent years, the fusion of LiDAR and camera by means of deep learning has attracted much attention and made great progress in 3D object detection. But feature receptive field of LiDAR points in 3D space and that of pixels in image plane are not consistent in size in deep learning process, this affects the LiDAR-Camera fusion quality. To alleviate this problem, we propose a cross fusion network in this paper to automatically choose image feature of appropriate receptive field to get fused with point cloud feature. Specifically, we input image features of all scales to the fusion module, and the weight of each image feature is calculated by our designed attentive point-image fusion module to control their contribution proportion. In addition, different from most point-based 3D object detection works, we adopt a semantic sampling strategy to replace commonly used farthest point sampling with assistance of image features, which retains more foreground points in the stage of point cloud down-sampling. We conduct experiments on KITTI and nuScenes datasetsand achieve advanced performance in 3D object detection benchmark. Comparative experiments with advanced fusion-based and point-based works demonstrate the excellent performance of our work.