Relation Inference Enhancement Network for Visual Commonsense Reasoning

When presented with a question regarding an image, Visual Commonsense Reasoning (VCR) offers not only a correct answer but also a rationale to justify the answer. Existing methods simply combine features from multiple modalities onto a shared dimension space, which doesn't align with human reasoning patterns, resulting in inadequate cross-modal and intra-modal reasoning behaviors. On the one hand, inadequate cross-modal reasoning arises from existing models relying on semantic correlations between answers and rationales in both textual modalities rather than the generative process of human reasoning from visual to textual modality. On the other hand, inadequate intra-modal reasoning arises from the incapacity of existing models to leverage previously acquired object relations beyond current observations like humans. To this end, we propose a novel Relation Inference Enhancement Network (RIE-Net), which enhances reasoning ability based on cross-modal image analysis and introduces intra-modal relational reasoning modules to memorize reasoning knowledge. To enhance the cross-modal association between images and rationales, RIE-Net introduces a cross-modal image analysis module, which eliminates language bias between answers and rationales by generating rationale from images. In addition, to comprehend and retain relational knowledge, RIE-Net introduces intra-modal relational reasoning modules to capture prior knowledge associated with various object categories and enhance the model's understanding of visual-spatial relationships. Quantitative and qualitative evaluations of the public VCR dataset demonstrate that our approach performs favorably against state-of-the-art methods.

I. Introduction

Visual Question Answering (VQA) refers to the task of answering natural language questions based on a given image. Although noticeable progress has been made, existing VQA benchmarks primarily focus on simple recognition questions (e.g., how many or what color) while neglecting the explanation of answering prediction. In light of this, the task of Visual Commonsense Reasoning (VCR) [1], [2] has recently been introduced to bridge the gap. Unlike traditional VQA that focuses on answering visual questions (Q A), VCR goes a step further by requiring the model to select the rationale behind the Q A process, which involves capturing visual commonsense (R), denoted as QA R.