Targetless LiDAR–Camera Extrinsic Calibration With Mesh-Based Constraints

Fusing multimodal information is prevalent in autonomous driving and robotics to conduct a variety of perception tasks. Light detection and ranging (LiDAR) sensors and cameras are complementary and the extrinsic parameters calibration between them is necessary for sensor data fusion. In this article, for the first time, we introduce mesh faces into the extrinsic calibration and propose a novel mesh-based calibration method for coarse-to-fine LiDAR-camera extrinsic parameter estimation, where the mesh is an efficient dense representation for LiDAR 3-D reconstruction. First, hand-eye calibration is utilized to obtain coarse extrinsic calibration parameters. Then, we formulate the extrinsic calibration as point-to-mesh constraints to refine initial parameters. Meshes are reconstructed from LiDAR point clouds and points are reconstructed using images. To speed up the calibration process, we propose to leverage ray tracing to achieve fast point-to-mesh data association. Additionally, we perform the observability analysis and Fisher information matrix (FIM) analysis to verify the feasibility of our method. Due to more accurate plane parameters provided by meshes than that by random sample consensus (RANSAC) using K-nearest neighbor (KNN) points, extensive experiments on both simulation and public real-world datasets show that our method achieves the best accuracies compared with state-of-the-art (SOTA) targetless methods. The average calibration error is 0.25° for the rotation error and 2.43 cm for the translation error on the KITTI dataset. Moreover, the proposed ray-tracing match method with more point-to-mesh matches is nearly 200 times faster than brute force matches with more matched pairs.