FusionNet: Coarse-to-Fine Extrinsic Calibration Network of LiDAR and Camera with Hierarchical Point-pixel Fusion

In this paper, we propose a novel network, Fusion-Net, which can estimate the extrinsic calibration matrix between LiDAR and a monocular RGB camera with high accuracy and robustness. FusionNet is a coarse-to-fine method, providing an online and end-to-end solution that can automatically detect and correct the decalibration without any specially designed targets or environments. First, the network applies deep-learning-based technologies to extract the features of LiDAR point clouds and RGB images. Then a novel method is adopted to fuse the features got from different sensors by projecting LiDAR features onto RGB feature maps, searching for the RGB features with the projected points as centers and concatenating the extracted RGB features with LiDAR features. To increase the accuracy, we apply a coarse-to-fine method in the network, by transforming LiDAR points and estimating the extrinsic calibration matrices from the coarse scale to the fine scale. The network is trained on random artificial decalibration matrices. Compared to existing approaches, our method doesn't need to train additional iterative networks, but it can also adapt to different ranges of decalibration.