Integrating Expert Knowledge with Vision-Language Model for Medical Image Retrieval

Content-Based Image Retrieval (CBIR) is an image search technique that can offer diagnostic guidance when facing difficult cases in radiology. State-of-the-art approaches propose to extract image features using vision-language models which learn image representations from supervision of text in medical literature. However, existing methods seldom take expert knowledge in medical domain into account. In this article, we propose a knowledge-and-language-guided contrastive visual representation learning framework for image retrieval. Our method consists of two steps: (1) modeling relationships between medical concepts and medical images using a knowledge graph, and translating each node in the graph into a knowledge embedding; (2) injecting knowledge embeddings into a vision-language model by aligning image representations using both encoded textual input and knowledge embeddings. Our experiments show that the proposed framework achieves comparable results to state-of-the-art methods on CBIR tasks using much less training data. Our code is publicly available at https://github.com/Wxy-24/KL-CVR.