Task-Specific Alignment Strategies for Enhancing Sentence Embedding

The pre-training and fine-tuning approach has become mainstream in natural language processing. While pre-trained models can generalize well, their performance can differ greatly across various domain tasks due to differences in data distribution. This indicates that different downstream tasks have different levels of sensitivity to sentence components. Our starting point is to scale Information of the model and data according to the specifics of downstream tasks, enhancing domain information of relevant tasks and reducing irrelevant elements for different domain tasks, called SIFTER. We use the SIFTER to improve Simple Contrastive Learning of Sentence Embeddings (SimCSE) by constructing positive sample pairs based on enhancing the sentence stem and reducing the unimportant components in the sentence, and maximize the similarity between three sentences. Our experiments demonstrate that SIFTER outperforms previous methods.