Incorporating Pre-trained Model into Neural Machine Translation

Since Transformer is proposed, it has boosted the neural machine translation baseline. In the past few years, although many researchers have tried to update the original transformer for each module, these approaches gain a little improvement. Pre-trained models have proved their power in natural language understanding tasks, how to incorporate the pre-trained model into natural language generation tasks such as machine translation is still an opening problem. This paper proposes a novel framework call Transformer-Pt,which combines the BERT and GPT-2 with Transformer. Experiment result on the WMT-14 dataset demonstrates our proposed method achieve improvement compared to other models.