Robust LiDAR-Camera 3-D Object Detection With Object-Level Feature Fusion

Cross-modal fusion methods enhance the 3-D object detection by leveraging multimodal complementarity. However, current methods, which seek to establish pointwise or region of interest-wise (RoI-wise) cross-modal correspondence, have their own limitations. The pointwise methods struggle with weak alignment and inaccurate depth estimation. The RoI-wise methods, which adopt RoI as the basis of sampling features and fusion features, produce ambiguous object-level features. In light of this, this article introduces object-level feature fusion network (OFFNet), a 3-D detector implementing cross-modal object-level feature fusion (COFF) discriminatively. To combine spatial information from peripheral light detection and ranging (LiDAR) features and unambiguous semantics from foreground image features, our approach operates fusion in two stages. The first stage unifies multimodal features into homogeneous representations in the point cloud space to facilitate cross-modal feature interaction. For this purpose, a point enhancement module (PEM) is designed to generate two sets of keypoints. A set of original keypoints are sampled from foreground LiDAR points to aggregate multidimensional LiDAR features in the COFF procedure. Meanwhile, a set of pseudokeypoints, generated by shifting the foreground keypoints toward the objects’ geometric center, are used as sampling points to aggregate foreground image features. In the second stage, we obtain robust object-level features by hierarchically fusing LiDAR and image features specific to each instance. Coarse 3-D candidate boxes are gridded and segmented into external and internal layers. The external layer aggregates LiDAR features from adjacent original keypoints, while the internal layer aggregates image features from neighboring pseudokeypoints. Experiments on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) and nuScenes benchmark demonstrate OFFNet’s state-of-the-art (SOTA) performance, particularly excelling in the challenging “cyclist” category.