Visual Robustness Benchmark for Visual Question Answering (VQA)

Can Visual Question Answering (VQA) systems maintain their performance when deployed in the real world? Or are they susceptible to realistic corruption effects, e.g., image blur, which can be detrimental in sensitive applications such as medical VQA? While linguistic robustness has been thoroughly explored within the VQA literature, there has yet to be any significant work on visual robustness. In this work, we present the first large-scale benchmark to evaluate the visual robustness of VQA models, including multimodal large language models and zero-shot evaluation, and assess the strength of the realistic corruption effects. Additionally, we have designed several robustness evaluation metrics that quantify an aspect of robustness. These sub-metrics can be aggregated into a unified metric and tailored to fit a variety of use cases. The experiments reveal important insights into the relationships between model size, accuracy, and robustness against the visual corruptions. Our benchmark highlights the need for a balanced approach in model development that considers model performance without compromising robustness.