Real-Time Orthopedic Surgery Procedure Recognition Method with Video Images from Smart Glasses Using Convolutional Neural Network

At present, orthopedic surgery has a large variety of surgical techniques. Procedures are complicated, and many types of equipment have been using in the surgery. So, operating room nurses who deliver surgical instruments to surgeon are supposed to be forced to incur a heavy burden. Although there is a navigation system for assisting surgeons in artificial joint replacement surgery, but no system exists for assisting operating room nurses. This work proposes a computer-aided navigation system that indicates the current procedure and procedure progress for nurses, and also instructs nurses to prepare surgical instruments to be used in the next procedure using smart glasses. Firstly, the system estimates the current status of the surgery procedure using a convolutional neural network (CNN) by utilizing real-time video images taken from smart glasses which was worn by operating surgeon. Then, the system indicates nurses the surgical instrument to be used at the next procedure in the smart glass worn by the nurses. The system was implemented with the object detection technology and the augmented reality. Experiment results demonstrated a satisfactory performance of our proposed system of recognizing surgery procedures.