Laparoscope Manipulating Robot (LMR) Navigation using Deep Learning-based Surgical Instruments Detection

During laparoscopic surgery, several items, including the laparoscope and all the surgical instruments must be manipulated at the same time. This paper presents the implementation of a surgical instrument detection system based on the deep learning methodology to work with the Laparoscope Manipulating Robot (LMR). Two procedures in this work are object detection on surgical instruments and then laparoscope navigation. The position information of detected surgical instruments is feed-backed to navigate the laparoscope automatically. The object detection comprises three main processes, the training, validating, and testing processes. Dataset in the experiments were gathered from 5 various public YouTube video clips related to gynecologic surgery. The previous work has compared various algorithms on surgical instrument detection. YOLOv4 provided the best performance with the validation result of the detection model across the experiment datasets with F1-score of 93.50% at the Confidence Threshold over 48% on the average during the training process. Hence YOLOv4 has been chosen to be assigned to the object detector. And as for the navigation process in the real-time surgical operation, once all surgical instruments have been detected, the focus of the laparoscope should be in the center among those instruments. The center position of each detected surgical instrument has been used to calculate the center of the laparoscope. To validate the viability of on-the-fly surgical instrument detection and laparoscope navigation, the proposed system has been implemented and tested on two test-cases of gynecologic surgery with the soft-tissue cadavers. The real tests have shown that the proposed method works well with two and three surgical instruments detection, but not with a single instrument detection, as the information on the position of a single surgical instrument detected is not enough to navigate the laparoscope to a designated object.