Breast3D: An Augmented Reality System for Breast CT and MRI

Adoption of Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) - known collectively as Extended Reality (XR) devices has been rapidly increasing over recent years. However, the focus of XR research has shown a lack of diversity in solutions to the problems within medicine, with it being predominantly focused in augmenting surgical procedures. Whilst important, XR applied to aiding medical diagnosis and surgical planning is relatively unexplored. In this paper we present a fully functional mammographic image analysis system, Breast3D, that can reconstruct MRI and CT scan data in XR. With breast cancer Breast Imaging-Reporting and Data System (BI-RADS) risk lexicon, early detection and clinical workflow such as Multi-disciplinary team (MDT) meetings for cancer in mind, our new mammography visualization system reconstructs CT and MRI volumes in a real 3D space. Breast3D is built upon the past literature and inspired from research for diagnosis and surgical planning. In addition to visualising the models in MR using the Microsoft HoloLens, Breast3D is versatile and portable to different XR head-mounted displays such as HTC Vive. Breast3D demonstrates the early potential for XR within diagnostics of 3D mammographic modalities, an application that has been proposed but until now has not been implemented.