In Vivo Demonstration of Deep Learning-Based Photoacoustic Visual Servoing System

Objective: To develop the first known deep learning-based photoacoustic visual servoing system utilizing point source localization and hybrid position-force control to track catheter tips in three dimensions in real-time. Methods: We integrated either object detection or instance segmentation-based localization with hybrid position-force control to create our novel system. Cardiac catheter tips were then tracked across distances of 40 mm in a plastisol phantom and 25-64 mm in an in vivo swine in real-time in nine visual servoing trials total. Results: Object detection-based localization identified the cardiac catheter tip in 88.0-91.7% and 66.7-70.4% of phantom and in vivo channel data frames, respectively. Instance segmentation detection rates ranged 86.4-100.0% in vivo. These catheter tips were tracked with errors as low as 0.5 mm in phantom trials and 0.8 mm in the in vivo trials. The mean inference times were ≥145.3 ms and ≥516.3 ms with object detection-based and instance segmentation-based point source localization, respectively. Hybrid position-force control system enabled contact with the imaging surface during ≥99.43% of each visual servoing trial. Conclusion: Our novel deep learning-based photoacoustic visual servoing system was successfully demonstrated. Object detectionbased localization operated with inference times that are more suitable for real-time implementations while instance segmentation had lower tracking errors. Significance: After implementing suggested optimization modifications, our novel system has the potential to track catheter tips, needle tips, and other surgical tool tips in real-time during surgical and interventional procedures.