Toward tool gesture and motion recognition on a novel minimally invasive surgery robotic system

Today's laparoscopic surgeon performs laparoscopic surgery by inserting two or three long tools into the patient's abdominal. Surgeon relies on the image captured by a laparoscope that is also inserted into the patient's abdominal. It is common to have a surgical assistant holding the laparoscope and taking directions from the main surgeon on how to move it to keep the surgical tools within the view. However, timely, accurate, and stable adjustments of the laparoscope cannot be guaranteed in lengthy operations due to fatigue of human assistant. We proposed a novel new design by replacing the human assistant with a spherical wrist mechanism robot. In addition, for developing a fully surgeon-based automatic integrated system, an automatic tracking and gesture recognizing sub-system is developed. The automatic tracking sub-system tracks the tip of a surgical tool and moves the laparoscope holding robot to keep the tool within the camera view. This paper focuses on the gesture recognizing sub-system that helps surgeon retrieve critical medical records of the patient instantly by recognizing surgical tool gestures. A modified background reduction method and neural network are applied in the tool gesture recognition process