Augmented Reality and Human–Robot Collaboration Framework for Percutaneous Nephrolithotomy: System Design, Implementation, and Performance Metrics

During percutaneous nephrolithotomy (PCNL) operations, the surgeon is required to define the incision point on the patient’s back, align the needle to a preplanned path, and perform puncture operations afterward. The procedure is currently performed manually using ultrasound or fluoroscopy imaging for needle orientation, which, however, implies limited accuracy and low reproducibility. This work incorporates augmented reality (AR) visualization with an optical see-through head-mounted display (OST-HMD) and human–robot collaboration (HRC) framework to empower the surgeon’s task completion performance. In detail, eye-to-hand calibration, system registration, and hologram model registration are performed to realize visual guidance. A Cartesian impedance controller is used to guide the operator during the needle-puncture task execution. Experiments are conducted to verify the system performance compared with conventional manual puncture procedures and a 2D monitor-based visualization interface. The results showed that the proposed framework achieves the lowest median and standard deviation error of 3.17 ± 1.36 mm in translation and 1.2 ± .9° in orientation across all of the experimental groups, respectively. Furthermore, the National Aeronautics and Space Administration task load index (NASA-TLX) user evaluation results indicate that the proposed framework requires the lowest workload score of 42.5 ± 13.7 for task completion compared to other experimental setups. The proposed framework exhibits significant potential for clinical application in the PCNL task, as it enhances the surgeon’s perception capability, facilitates collision-free needle insertion path planning, and minimizes errors in task completion.