Reinforcement Learning-Based Ultrasound Probe Navigation for Magnetic Guidewire Tip Tracking in Vascular Environment

Ultrasound(US) imaging, owing to its non-invasive, real-time and cost-effective characteristics, holds great potential in autonomous vascular interventional procedures. During such procedures, ultrasound images are typically used to estimate the position of the magnetic guidewire, providing essential guidance for accurate delivery. However, a fully automated ultrasound-based guidewire tracking solution has not yet been established. In this work, we propose a reinforcement learning-based two-stage magnetic guidewire tip tracking framework. A YOLOv11-based module is employed to extract visual features from ultrasound images, thereby reducing the complexity of model training, while a DQN agent is used to control translational movements of the probe. Additionally, probe rotation is controlled using preoperative vascular information. All RL agents are trained entirely within a SOFA-based simulation environment, significantly reducing training resource costs. The effectiveness of the proposed framework was demonstrated on a femoral artery phantom, with tip tracking rates reaching 54.71±6.12% in simulation and 49.88±7.26% in real-world experiments.