Intraoperative visual guidance and control interface for augmented reality robotic surgery

We proposed an augmented reality (AR) robotic system equipped with intraoperative visual guidance and gesture based control interface. The proposed feature is an enhancement of our AR robotic system. AR robotic system was introduced to the field of interventional medicine to assist surgeons in implementing medical operations under the augmented reality environment. The technique combining dextrous robot and AR guidance provides a new operational mode for surgeons. The introduction of robotic modules is to compliment surgeon's dexterity and to perform specific tasks defined during the surgery. Augmented reality provides additional visualization and interaction absence in a typical clinical environment. The interfacing system includes intraoperative tracking of surgical tools and virtual reconstruction of the visually occluded tool segment, and an intuitive gesture-based human-computer interaction centered on the projected organ. Preliminary experiments show that this novel human-machine interface is effective for surgical intervention.