An Efficient Method of Supervised Contrastive Learning for Natural Language Understanding

Fine-tuning on the pre-trained language models has achieved state-of-the-art performance in many natural language understanding (NLU) tasks. For classification tasks, the cross-entropy loss is commonly employed as optimized objective in the fine-tuning stage. However, models fine-tuned with this method can not gather the contrastive information between positive and negative samples. The data points encoded by these models are close to decision boundary. In this paper, we propose a supervised contrastive learning (SCL) method to better fine-tune in text classification and regression tasks. The proposed SCL loss obtains a large margin. In addition, we explore 6 text augmentation strategies to generate hard positives and negatives at the embedding layer. The ELECTRA-Base and ELECTRA-Large fine-tuned with our method surpass the MPNet and DeBERTa-Large on the GLUE validation set, respectively. Experimental results show that this method leads to better generalizability, with 10.1 points improvement on 3 datasets of GLUE in zero-shot settings. The code is available at: https://github.com/littlewhite-gao/SCL.