Contrastive Invariant Risk Minimization for Grounded Situation Recognition

Grounded situation recognition (GSR) is a comprehensive structured scene understanding task that predicts the salient activity (verb), entities (nouns) involved in the activity with their roles, as well as the corresponding bounding-box groundings of the entities from the given image. Existing I.I.D.-based methods for GSR are limited in their ability to recognize novel verb-noun combinations. To address this problem, in this paper, we novelly consider GSR as a Non-I.I.D. task and focus on learning verb-invariant and role-specific representations for verb and noun predictions. Based on the causality, a novel Contrastive Invariant Risk Minimization (CIRM) model for GSR is proposed. In the proposed CIRM, invariant risk minimization is integrated into a transformer architecture to learn invariant representations for verb prediction. To enhance the intra-verb compactness and the inter-verb separability, contrastive learning is utilized to learn discriminative features. As far as we know, this is the first work that regards the task of GSR as a problem of out-of-distribution generalization. Extensive experiments on the benchmark SWiG dataset demonstrate the effectiveness of our proposed CIRM over other state-of-the-art methods in all evaluation metrics.