Recombination Samples Training for Robust Natural Language Visual Reasoning

Natural Language for Visual Reasoning (NLVR) is a challenging binary classification reasoning task, requiring the model to determine whether a textual description is true about a pair of images. However, in this task, if the part of input content (such as only color or quantity) is changed, the recent model is difficult to make the correct judgment. To this end, we propose a novel Recombination Samples Training (RST) scheme for increasing model robustness with stronger reasoning ability. Specifically, the RST synthesizes a lot of unlabeled new samples through the recombination and uses entropy minimization loss to train them. These samples through different combinations of the same content make the model focus on reasoning logic rather than some superficial correlations. Moreover, the RST calculates the consistency loss between the model trained with and without new samples to prevent overfitting. Experiments in the NLVR2 dataset demonstrate the effectiveness of the method.