Keypoint-Based LiDAR-Camera Online Calibration With Robust Geometric Network

There is no denying the fact that LiDAR-camera online calibration is an extremely complex problem. For automatic driving technology development, a stable, accurate, and fast online LiDAR-camera calibration system becomes quite essential. As deep learning plays a vital role in many fields, learning-based methods promise to solve this problem. However, the existing methods do not fully consider the geometric constraints in the calibration process. In this study, we propose a new network: RGKCNet, a 2-D–3-D pose estimation network based on keypoints, which can be applied to calibrate the camera and LiDAR in real time. Using the declarative layer, extrinsic calibration is regarded as a bilevel optimization problem, which allows us to embed an optimizer based on geometric constraints in the end-to-end network and, thus, realize 2-D–3-D data association. With a trainable point weighting layer, the network can extract sparse keypoints and give the corresponding weights used in pose estimation, which further improves the robustness of the network. We verify the performance of the proposed method on the KITTI dataset to demonstrate the significant effect in practical applications.