Multimorbidity Content-Based Medical Image Retrieval and Disease Recognition Using Multi-Label Proxy Metric Learning

Content-based medical image retrieval is an important diagnostic tool that improves the explainability of computer-aided diagnosis systems and provides decision-making support to healthcare professionals. A common approach to content-based image retrieval is learning a distance metric by transforming images into a feature space where the distance between samples is a similarity measure. Proxy metric learning methods are effective at learning this transformation due to the use of proxy feature vectors that enable efficient learning. Training with a distance-based classification loss enables a single proxy model to be suitable for both retrieval and classification. However, these methods are designed only for single-label data, making them unsuitable for multimorbidity medical images. Addressing this, we propose a novel multi-label proxy metric learning method for content-based image retrieval and classification. Unlike existing proxy-based methods, training samples assign to multiple proxies that span multiple class labels. This results in a feature space that encodes the complex relationships between diseases. We introduce negative proxies to better encode the relationships between samples without detected diseases. The efficacy of our approach is demonstrated experimentally on two multimorbidity radiology datasets. Results show that our method outperforms state-of-the-art image retrieval systems and baseline approaches. Our method is clinically significant as it improves on two key factors shown to affect medical professionals’ willingness to use computer-aided diagnosis systems: accuracy and interpretability.