A Self-Supervised Generative Fine-Tune of CLIP for VQA for Visually Impaired

Vision-language models like Contrastive Language-Image Pre-Training (CLIP) have significantly progressed in multi-modal tasks such as scene description and visual question answering (VQA). A similar task can be solved using large visual models (LVMs), or state-of-the-art Retrieval Augmented Generation (RAG) methods. The primary challenges of such systems are the training cost and data-hungry nature. At the same time, small models such as CLIP can be fine-tuned for specific tasks, but they also demand sufficient training data, which can be difficult in many scenarios. This article proposes a novel self-supervised learning of a VQA system customized for the visually impaired (VI). We have shown that a pre-trained VQA model can be fine-tuned for a VQA system designed for VI people without training ground truth. We utilize an optical character recognition (OCR) module to extract the text. Next, another pre-trained question-answer generative module generates a synthetic question-answer pair. Finally, the synthetic question-answer is used to fine-tune the CLIP module. We found that the proposed method outperforms CLIP by a reasonable margin. The code will be available here.