Image-to-Label-to-Answer: A Simplified LLM Framework Medical Visual Question Answering

Medical Visual Question Answering (MVQA) has emerged as a promising approach for assisting clinicians in interpreting complex medical images by providing accurate and context-aware answers to specific visual queries. In this project we introduce an innovative framework, "Image to Label to Answer" (ILA) designed to enhance the efficiency and accuracy of clinical applications in MVQA. The proposed project will involve a setup of integrated image recognition techniques along with advanced natural language processing models that systematically convert the medical images into interpretable labels, which in turn generate the correct answers to clinical questions. The ILA framework will operate in three stages: image labelling, context generation, and answer synthesis. First, the system uses deep learning-based image segmentation and classification models to extract relevant features and assign diagnostic labels to medical images. These labels provide the basis for generating contextual understanding, which is crucial for the subsequent stage. In the context generation phase, the framework uses ontology-based reasoning in combination with language models to interpret the labelled data within the clinical context. As seen, the synthesis module can build full and accurate replies after the contextual understanding created over the input queries.