Synergistic Digital Twin and Holographic Augmented-Reality-Guided Percutaneous Puncture of Respiratory Liver Tumor

Thermal ablation is an exciting new minimally invasive treatment that destroys liver tumors without removing them. It uses image guidance to place a needle through the skin into a liver tumor, which is highly dependent on surgeons’ experience. With the development of digital medicine, augmented reality (AR) has become a more intuitive and safer way to achieve real-time navigation. However, the technology is still in its infancy due to its limited accuracy and real-time performance. To address these problems, we syncretized the holographic AR with the digital twin technique to track the dynamic surgical scene and provide the 3-D navigation of heterogeneous target regions via internal motion prediction. To tackle the dilemma of real-time performance and precise internal motion estimation, a dynamic adaptation scheme is proposed to compensate for the time cost induced by the external/internal correlation model and data transmission. We carried out a series of experiments to validate our methods. With the proposed external/internal correlation model, the average estimation errors of the tumor and vessels are 2.18 and 2.79 mm, respectively. Besides, we performed in vivo experiments on two beagle dogs with an artificial lesion in their liver, respectively, and the puncture accuracy of our method are 2.5 and 2.17 mm. The results show that on one hand, our method can fulfill the real-time requirement of AR via using the intraoperative data, which is also more precise than that with preoperative data. On the other hand, our method can provide more 3-D information for surgeons, such as vessels, which can well ensure the safety of operation.