Cross-Lingual NLP: Bridging Language Barriers with Multilingual Model

In the case of multilingual sequence-to-sequence pretrained language models (multilingual Seq2Seq PLMs), such as mBART, the self-supervised pretraining task is trained on a variety of monolingual tongues, such as the twenty-five languages from CommonCrawl1, while the downstream cross-lingual tasks normally advance on a bilingual communication subset, such as English-German, that result in a discrepancy among the pretraining and fine-tuning phases with regard to of both data and cross-lingual. We add an additional codeswitching restore task to the vanilla pretrain-finetune pipeline in order to fill in the aforementioned cross-lingual domain and task gaps. In particular, the self-supervised code-switching restoration job is used as a pretext task in the first step so that the multilingual Seq2Seq PLMs can learn some indomain alignment information. In the subsequent phase, we adjust the model using downstream data as usual. Our framework continually outperforms the strong baseline mBART using standard finetuning strategies, as evidenced by testing on both NLG (12 bilingual transformation tasks, 30 zero-shot transformation tasks, and 2 cross-lingual summarise tasks) and NLU (7 cross-lingual language deductive reasoning tasks). Results show that our method might enhance the model's generalisation at negligible computational cost and reduce the Euclidean distance across cross-lingual sentence representations.