HL-SfM: A Marker-free and Automated Hologram Registration and Tracking Framework for Image-guided Surgery

The application of mixed reality (MR) technology is highly attractive for the emerging concept of image-guided surgery, as MR is capable of providing the surgeons with overlaid information of anatomical structure and navigation information, thus leading to more accurate surgical decision making. The accuracy of the overlaid information is dependent on the registration and tracking of holograms. Nevertheless, current registration methods either suffer from insufficient accuracy or require complex preparation, hindering their direct translation to a clinical scenario. To address these, we propose ’HL-SfM’, a marker-free, fully-automated, and high-precision hologram registration and tracking framework based on the 3D reconstruction with an optical see-through MR device–HoloLens2. Specifically, we use a series of obtained RGB images for target region matching, followed by real-scale scene reconstruction via high-precision initial SLAM data. Afterwards, we apply point clouds-based registration algorithm for hologram alignment. Additionally, we propose a novel "detect-LoFTR" module, an efficient target feature matching method, for feature selection in the 3D reconstruction process. In the end, HL-SfM achieves an average error of 2.6 mm and 2.4°, outperforming the results directly obtained from the point clouds of the HoloLens2 AHaT camera, which is also state of the art for marker-free registration in HoloLens2-only methods. Our supplementary materials can be found at Baidu Netdisk or Google Drive.