Generate to Understand for Representation in One Pre-training Stage

A lot of high quality pretrain works are introduced in recent and take magnificent influence in Understanding (NLU), Generation (NLG) and Representation of Natural Language Processing tasks. tasks. In the traditional paradimg, the models are pretrained on custom domain corpus and finetuned on specific tasks, which make them very expensive for gpus and workers. What is worse, recent trends in language modeling is focusing on increasing performance through scaling with a booming cost.We propose a simple and efficient pretraining framework GUR that unifies the language modeling and contrastive learnin tasks in one pretraining step. Based on the examples of similar text pairs chosen by their LCS (longest common string) from raw unlabeled documents, our pretraining scheme consists of masked language modeling combined with a unsupervised contrastive learning task. The resulting model GUR performs surprisingly well without any labeled training datapoints. Specifically, it significantly outperforms all other pretrained baselines as a retriever at the recall benchmark in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Of course, the GUR keeps the ability of language modeling in our ablation experiment. We release our code https://github.com/laohur/GUR.