A New Precise Contactless Medical Image Multimodal Interaction System for Surgical Practice

Blind spot confirmation as a major challenge in surgical practice has led to booming of 3D modeling and printing technology and applications, but existing technologies could not offer readily available service of medical images in the operation room (OR) for surgeons. Contacting with keyboards and mice or asking assistants to help in viewing medical images during the operation could increase operation risk, infection and prolong of the operation time. This paper presents a new precise contactless medical image multimodal interaction system for surgeons in the OR. By supporting seamless multimodal interaction via 2D laser positioning, 3D gesture and voice recognition, it overcomes the design flaws in other similar solutions such as big motion for gesture recognition, round trip towards the screen of surgons, difficulty in selection and comparison of images, and more. Following the system engineering and evidence-based approaches, the design and development of this new precise contactless multimodal medical image interaction system meet surgeons' real world requirements for efficient and effective 2D medical images and 3D model operation under both preoperative and intraoperative scenarios. The system evaluation study is carried out in conjunction with a group of 10 neurosurgeons with 3 to 10 years' of surgical experience from different hospitals. Evaluation results successfully demonstrate the system's feasibility, efficiency and usability of in surgical contexts.