Targetless LiDAR-Camera Extrinsic Calibration with Mesh-Based Constraints

Fusing multi-modal information is prevalent in autonomous driving and robotics to conduct a variety of perception tasks. Light Detection and Ranging (LiDAR) sensors and cameras are complementary and the extrinsic parameters calibration between them are necessary for sensor data fusion. In this paper, for the first time, we introduce mesh faces into the extrinsic calibration and propose a novel mesh-based calibration method for coarse-to-fine LiDAR-camera extrinsic parameter estimation, where mesh is an efficient dense representation for LiDAR 3D reconstruction. First, hand-eye calibration is utilized to obtain coarse extrinsic calibration parameters. Then, we formulate the extrinsic calibration as point-to-mesh constraints to refine initial parameters. Meshes are reconstructed from LiDAR point clouds and points are reconstructed using images. In order to speed up the calibration process, we propose to leverage ray tracing for achieving fast point-to-mesh data association. Additionally, we perform the observability analysis and Fisher Information Matrix (FIM) analysis to verify the feasibility of our method. Due to more accurate plane parameters provided by meshes than that by Random Sample Consensus (RANSAC) using K-Nearest Neighbor (KNN) points, extensive experiments on both simulation and public real-world datasets show that our method achieves the best accuracies compared with state-of-the-art targetless methods. The average calibration errors is 0.25° for rotation error and 2.43cm for translation error on KITTI dataset. Moreover, the proposed ray tracing match method with more point-to-mesh matches is nearly 200 times faster than brute force matches with more matched pairs.