Distance Metric Learning-optimized Attention Mechanism for Visual Question Answering

Visual Question Answering (VQA) is a multimodal task that answers questions by exploring the content of the images. A comprehensive understanding of both the images and questions is critical for VQA. Although great success has been achieved in VQA recently, the problem of language bias hinders the application of VQA models in real-life situations. Language bias leads to the inability to generalize the VQA model to test sets with different distributions. Therefore, we propose a distance metric learning optimized attention mechanism (DMA), which selects and directly adopts the image information that is related to the question, performing feature selection and compression via a novel unsupervised metric learning scheme. Specifically, we only keep the image features with higher similarities with the question to answer the question. We adopt unsupervised metric learning to separate image features with high similarity to the question from those with low similarity to the question in the feature representation space. By doing this, the difficulty of VQA models understanding of images is reduced and language bias is mitigated. Extensive experiments have proved the effectiveness of our method for both Vanilla Transformer and Cross Transformer, with improved results on VQA v2 and VQA-CP datasets. Particularly, by building on top of model LMH+CSS, we achieve State-of-the-art performance on VQA-CP datasets.