Augmented Reality Based Surgical Training and Education System for Neurosurgery

Surgical simulation helps trainees to explore many clinical scenarios before experiencing them with patients. Augmented reality (AR) based on patients’ medical data is an innovative technique for medical education and neurosurgical practice outside the operating room. We propose an AR application for surgical training of pituitary tumor resection in order to help trainees understand different conditions of pituitary tumors and nearby vessels, especially arteries. Virtual objects of pituitary tumors and nearby arteries and veins are created from the image segmentation of multiple patients’ MRI data. To provide an immersive learning environment, we develop a gesture-based user interface based on a head-mounted display HoloLens2 which allows trainees to manipulate the visualization of virtual objects in real time, such as zooming in/out, panning and rotating the virtual objects. The proposed AR application can be used to simulate pituitary tumor resection at different levels of difficulty based on our database which contains various virtual models of pituitary tumors and the nearby vessels.