ICT-QA: Question Answering Over Multi-Modal Contexts Including Image, Chart, and Text Modalities

For question answering in multi-modal contexts that include image, chart, and text modalities, a model must be proficient in understanding each individual modality. Furthermore, the model must be able to find the necessary evidence from multiple modalities and generate answers through cross-modal reasoning for some questions. In this paper, we propose the Image and Chart Instruction Tuning (IC-tuning) method to enhance the model's comprehension of each modality. Specifically, we introduce visual-aware chart instruction-following data that describe both precise numerical values and visual information on the charts. We then train a Large Language Model (LLM) with a model architecture that utilizes an image-specific encoder and a chart-specific encoder. Our experiments demonstrate that this method achieves state-of-the-art performance in Chart Summarization and Open-ended Chart question answering (OpenCQA) tasks while having minimal impact on image and language benchmark performance. Although the IC-tuned model shows great comprehension performance for each modality, it still struggles with question answering tasks in multi-modal contexts because it is only trained on data for understanding each individual modality. To address this, we introduce the Question Answering over Image, Chart, and Text (ICT-QA) dataset, designed specifically for question answering in multi-modal contexts. After further training the IC-tuned LLM with the ICT-QA dataset, our evaluations demonstrate that ICT-QA significantly improves the quality of answers for both single-modal questions, where only one modality needs to be referenced from multiple modalities, and cross-modal questions, which require reasoning across multiple modalities.