A Weak Supervision-based Robust Pretraining Method for Medical Visual Question Answering

Medical images are complex, and the annotation of medical images requires high expertise. It would be time-consuming and costly to annotate directly for experts. As a result, one of the primary challenges currently faced by medical visual question answering (VQA) is the lack of large-scale annotated data. To address this issue, a Weak Supervision-based Robust Pretraining (WSRP) method for medical VQA is proposed. Specifically, our method builds upon a contrastive language-image pretraining framework by introducing adversarial training. However, the contrastive language-image pretraining framework, by treating each image-text pair as a separate category, may lead to class collision problems, thereby affecting the quality of image representation. Therefore, weakly supervised contrastive learning is introduced to generate weak labels, enabling the model to learn fine-grained feature representations. The proposed weak supervision-based robust pretraining method for medical VQA is empirically evaluated and experimental results on public datasets demonstrate its superior performance.