Forceps Motion Estimation in Laparoscopic Surgery using Deep Learning for Robotic Camera Control

In laparoscopic surgery, the laparoscope is controlled by a camera assistant according to verbal instructions from the surgeon. A laparoscope holder is effective to provide stable view. However, the surgeon must control the holder. The autonomous control of the holder is helpful to the surgeon to focus on the surgery. The forceps tip in laparoscopic image becomes the target of the automatically controlled robot. Estimating the future movement of the forceps can improve the control performance of the holder robot. In this paper, we propose a method to estimate the forceps tip motion 0.1 seconds ahead by deep learning using the segmented forceps in camera image. The center of gravity of the gripper in the forceps was calculated from the segmented image. The future coordination of the forceps gravity was estimated by deep learning using past gravity positions. The neural network structure of deep learning was determined by comparing computational errors. We experimentally confirmed that the forceps movement of 0.1 seconds ahead can be estimated online in pick-and-place and suture tasks.