Enhancing Sentence Representation with a Transfer Task Enhancement Auxiliary Network

Contrastive Learning of Sentence Embeddings (CSE) has gained significant traction in the realm of Natural Language Processing (NLP), proving to be particularly effective in applications like sentence similarity assessment and text retrieval, where supervised methods are currently achieving the best performance. We have identified a trend: although supervised sentence representation methods show remarkable effectiveness in semantic similarity assessments, their performance tends to decline in transfer tasks. In order to solve this problem, we propose to use transfer task enhancement auxiliary network, named TEAnet. Our method leverages rationales generated by large language models to guide smaller models, specifically by jointly conducting classification loss training during the contrastive training process. Our experiments demonstrate that the auxiliary network approach enhances performance on the transfer task while maintaining the semantic similarity task of training.