Quantity-Aware Coarse-to-Fine Correspondence for Image-to-Point Cloud Registration

Image-to-point cloud registration aligns the data acquired by heterogeneous sensors such as camera and LiDAR. To capture cross-modal correspondences for estimating the relative pose, matching points with pixels presents great ambiguity due to the disparate visual attributes. This article focuses on high-level semantic matching between point sets and pixel patches to alleviate the ambiguity, while accurately refining the coarse results to the point and pixel levels. However, it is impossible to generate perfectly aligned point sets and pixel patches when self-organizing the elements without camera pose priors, thus introducing numerous outliers during fine-level matching. Therefore, we propose a learning framework to bind the quantity of points with the degree of semantic correlation between point sets and pixel patches. The learned quantity-aware correspondences play two critical roles: indexing pixel patches for each point set based on their degree of correlation and estimating the number of outliers from point sets to pixel patches. Specifically, we learn local semantic proxies to estimate the cost of the optimal transport problem from point sets to pixel patches, where the points are viewed as allocatable entities and patches as receptacles. A novel supervision strategy is then proposed to quantify the degree of correlation as continuous values. Afterward, point-to-pixel correspondences are refined from a smaller search space by a resample-then-filter scheme, with a confidence sorting strategy to adaptively remove outliers based on the quantity-aware priors. Extensive experiments on the KITTI Odometry and NuScenes benchmarks demonstrate the superiority of our method over the state-of-the-art methods.