Cross-modal LiDAR-Visual-Inertial Localization in Pre-built LiDAR Point Cloud Map through Direct Projection

This paper proposes the LiDAR Map LiDAR-Visual-Inertial Localization (LM-LVIL) system, integrating LiDAR, visual, and inertial sensors within a pre-constructed LiDAR map. The key innovation is cross-modal fusion between camera images and point cloud maps via direct projection. Unlike traditional methods aligning 3D points triangulated from camera with the point cloud, our system projects map points into the camera frame to minimize photometric error, similar to direct methods in visual Simultaneous Localization and Mapping (SLAM). The system uses an iterated error state Kalman filter (IESKF) framework, incorporating IMU (Inertial Measurement Unit) integration, visual measurements, and LiDAR measurement for precise pose estimation. LiDAR measurement aligns points by ensuring scan points lie on a local plane in the map. Visual measurements include photometric measurement, projection measurement, 3D point alignment, and 2D-3D line measurement. Experimental results show that LM-VIL (LiDAR Map Visual-Inertial Localization) achieves competitive accuracy indoors compared to state-of-the-art methods. The LM-LVIL system improves accuracy in both indoor and outdoor scenarios compared to the LM-LIL (LiDAR Map LiDAR-Inertial Localization) system and the LiDAR localization baseline method. Our localization system balances accuracy and real-time performance, demonstrating application potential.