Aggregating embeddings from image and radiology reports for multimodal Chest-CT retrieval

This paper proposes a multimodal retrieval system for Chest CT (called ChestFinder) that combines image and report embeddings’ into a filter-and-query strategy. ChestFinder is composed of three modules, namely (i) text transformation, (ii) feature extraction, and (iii) ranking aggregation. Text transformation is conducted by a fine-tuned Generative Pre-training Transformer model (GPT), and the image embeddings are extracted after (i) training a Residual Neural Network (ResNet-50), and (ii) reducing and scaling the encoded vectors. ChestFinder produces one list of similar images and another of related reports for each query input composed of a Chest CT with a radiology report. Then, the ChestFinder ranking aggregation module fuses those two lists to produce the final ordered set of retrieved objects, as in a top-k query. The aggregation is performed by a fine-tuned Threshold Algorithm (TA) whose weights are calculated by a Multi-Layer Perceptron (MLP) trained to label the reports. To examine the quality enhancement brought by this multimodal search, we constructed a dataset of Chest CTs from our University Hospital PACS/RIS systems by filtering distinct cases diagnosed with emphysema (one finding per case and with at least two radiologists agreeing on the diagnosis). A holdout experimental evaluation showed the ChestFinder search achieved higher Accuracy and Sensitivity than content-only top-k searches. Results also indicated quality gains drawn from the adjustments of ChestFinder modular components: (i) fine-tuned GPT achieved up to 0.89 F1-Score in data testing with a stable train/validation ratio for radiology reports, (ii) fine-tuned GPT significantly outperformed the zero-shot approach as well as a fine-tuned BERT, (iii) non-weighted ranking aggregation increased the search accuracy in up to 10%, and (iv) fine-tuned TA outperformed the baseline and non-weighted ranking aggregation in up to 52%.