Deep Learning Assisted Robotic Magnetic Anchored and Guided Endoscope for Real-Time Instrument Tracking

This letter presents the first case of implementing deep-learning based instrument tracking on a magnetic anchored surgical endoscope. The compact magnetic actuated endoscope has a unique structure that allows operations near the anchor surface, ideal for video assisted thoracoscopic surgery (VATS). Autonomous tool tracking alleviates surgeon's burden and prevents human errors from muscle fatigues or miscommunication. However, conventional methods rely on color labels or require modification to instrument, and has risk of failure due to occlusion of marker. In this letter, we combine deep-learning instrument detection with visual servoing control. This allows the magnetic endoscope to track surgical tools automatically, without color markers or instrument modification. We used a modified TernausNet-16 network that can detect surgical instrument in real time, with a small training dataset of 1846 images. Experiments show that the magnetic endoscope can effectively track a marker-less instrument. It can also track continuous motions of a target traveling at 40 mm/s. The performance was also verified by completing mock-up surgical task in a simulated thoracic cavity.