Real-time 2D video/3D LiDAR registration

Progress in LiDAR scanning has led to the availability of large scale LiDAR datasets for urban areas. We use such pre-acquired data to determine the poses of 2D monocular cameras highly accurately in real-time. This is achieved by first correctly aligning key-frames of the multi-modal data using a combination of feature and intensity-based 2D/3D registration methods. The online pose is then determined in realtime by densely sampling and tracking features within the 2D video stream. The 3D coordinates of these features are determined by a fast GPU-based backprojection. The observed 2D/3D feature data is then fused using a recursive Bayesian filter in order to exploit temporal coherency. The method is evaluated using ground truth camera trajectories and different filter implementations. The proposed registration and filter framework executes at video-frame rate and it is up to 15% more accurate then a registration only solution. Applications are numerous and include, for instance, augmented-reality applications, online geo-referentiation or metric online 3D reconstruction from monocular video data.