Disease-Oriented Image Embedding With Pseudo-Scanner Standardization for Content-Based Image Retrieval on 3D Brain MRI

To build a robust and practical content-based image retrieval (CBIR) system applicable to clinical brain MRI databases, we propose a new framework, disease-oriented image embedding with pseudo-scanner standardization (DI-PSS). It consists of two core techniques: data harmonization to absorb differences caused by different scanning environments and an algorithm to generate low-dimensional embeddings suitable for disease classification. Until now, there have been very few studies aimed at CBIR of brain MRI. Even in the harmonization of scanners, which is an important prerequisite technique for CBIR, only a limited number of studies have been conducted on T1-weighted MRI, which has collected a vast amount of clinical data. Recently proposed methods need to correctly estimate the domain (i.e., dataset, scanner) of each data in advance to remove environment-dependent information from low-dimensional embedding, which is not an easy task. With DI-PSS, each brain image is pseudo-transformed into a brain image taken with a given reference scanner. Then, 3D convolutional autoencoders (3D-CAE) trained with deep metric learning generate low-dimensional embeddings that better reflect the characteristics of the disease. In this study, DI-PSS reduced the variability of distance in low-dimensional embedding between Alzheimer’s disease (AD) and clinically normal (CN) patients, caused by differences in scanners and datasets, by 15.8-22.6% and 18.0-29.9%, respectively, compared to the baseline. This improved the ability of spectral clustering to classify AD and CN by 6.2% in average accuracy and 10.7% in macro-F1. Our method has the advantage of not requiring difficult domain prediction tasks in advance, and can effectively utilize the big data of T1-weighted MR images. Given the potential of the DI-PSS for harmonizing images scanned by MRI scanners that were not used to scan the training data, it is well suited for application to a large number of legacy MRIs captured in heterogeneous environments.