Framework for Deep Learning Based Multi-Modality Image Registration of Snapshot and Pathology Images

Multi-modality image registration is an important task in medical imaging because it allows for information from different domains to be correlated. Histopathology plays a crucial role in oncologic surgery as it is the gold standard for investigating tissue composition from surgically excised specimens. Research studies are increasingly focused on registering medical imaging modalities such as white light cameras, magnetic resonance imaging, computed tomography, and ultrasound to pathology images. The main challenge in registration tasks involving pathology images comes from addressing the considerable amount of deformation present. This work provides a framework for deep learning-based multi-modality registration of microscopic pathology images to another imaging modality. The proposed framework is validated on the registration of prostate ex-vivo white light camera snapshot images to pathology hematoxylin-eosin images of the same specimen. A pipeline is presented detailing data acquisition, protocol considerations, image dissimilarity, training experiments, and validation. A comprehensive analysis is done on the impact of pre-processing, data augmentation, loss functions, and regularization. This analysis is supplemented by clinically motivated evaluation metrics to avoid the pitfalls of only using ubiquitous image comparison metrics. Consequently, a robust training configuration capable of performing the desired registration task is found. Utilizing the proposed approach, we achieved a dice similarity coefficient of 0.96, a mutual information score of 0.54, a target registration error of 2.4 mm, and a regional dice similarity coefficient of 0.70.