Temporal-spatial validation of knot-tying procedures using RGB-D sensor for training of surgical operation

We propose a method for validating surgical knot-tying motions for beginner's self-training. Our system observes trainee's hands by a RGB-D sensor and describes the point cloud as a SHOT feature. The features are used for matching an input image sequence to that of an expert by dynamic programming. The type of the knot-tying motion is recognized by the matching score. The matching enables the system to recognize type of input knot-tying motion and to validate each frame. The system specifies inappropriate frames by temporal validation based on the difference of the matched frames. Then the system detects and shows inappropriate parts in each inappropriate frame by utilizing the spatial structure of the feature. We tested our method on a motion dataset of novice trainees and achieved favorable performance.