Human Interface and Control of a Robotic Endoscope Holder Based on an AR Approach

This paper proposes an augmented reality (AR) system that facilitates surgeons to operate a robotic endoscope holder more intuitively. Through the use of an AR glasses, a surgeon can observe the endoscope view and at the same time perceive the condition of the operation instead of completely isolating from the real world, and thus also increase safety. Further, in this robotic endoscope control system, a surgeon can directly use his/her head movement to control the endoscopic view. Since the head-control approach cannot directly handle the depth of the endoscope, we propose a depth control scheme based on instrument identification. In order that the automatic depth adjustment will not interfere with the MIS operation, the system will first determine whether the surgeon is performing the operation or not. The depth control of the endoscope is allowed to work only when the surgeon is not performing the surgical instrument movement. Practical experiments using a HIWIN 6DOD robotic arm and Epson BT300 AR glasses show that the user can effectively control the movement of the endoscope to complete a tracking task of the surgical instrument.