Analysis of Multimodal Representation Learning Across Medical Images and Reports Using Multiple Vision and Language Pre-Trained Models

Medical images are the visual representation of the organs and their functions. The medical images are used for finding the diseases and highly trained professionals interpret these images into reports. The interpretation of these images requires a lot of time in a day for the reports summarization and highlighting the findings for the naive person. Enhancing the medical diagnosis of diseases through the generation of summarized reports along with findings has immense potential. The sensitivity of the findings of the chest x-rays provided by the existing techniques are not so accurate and the limitation of data also affects the training of the models. Therefore, a joint image-text embedding extraction from medical images and reports, in radiology reports summarization along with correct findings will significantly assist the doctors and reduce their burden. In this approach, learning from multimodal representation using radiographs and reports through pre-trained vision-and-language models. The pre-trained model classified the diseases or findings in the chest x-rays and used Gated Recurrent Units to generate a summarized report. The image-text joint embedding using the pre-trained models helps in more accurate report summarization and improves performance in thoracic findings tasks.