Toward Enhancing Cross-Lingual Domain Knowledge Sharing and Transferring for Multilingual Domain Adaptation in NMT

Multilingual Neural Machine Translation (NMT) excels in sharing knowledge across languages and transferring insights from high-resource languages to improve performance in low-resource languages. However, its performance lags in specific domains such as legal and medical. Previous works have focused on adding language-specific and domain-specific adapters to achieve domain adaptation. Although effective, these adapter-based methods only use domain data to train additional parameters, limiting the performance of multilingual NMT. In this paper, we propose CDSTX, a novel approach that achieves robust multilingual domain adaptation without relying on adapters, solely leveraging multilingual models. Specifically, we utilize the pretrained model XLM-R and frozen embeddings to preserve its robust multilingual capabilities and design a two-stage training strategy for domain adaptation. This includes separate training for the decoder and fine-tuning of the entire model, ensuring that the model effectively acquires domain knowledge and correctly represents domain-specific text. Moreover, to better utilize the domain information conveyed implicitly by the training data, we devise special domain tokens at the beginning of the source and target sentences called Source&Target Domain Tags. In addition, back translation is employed to enhance the cross-lingual transfer ability of our approach. Our proposed method is evaluated on two datasets across three translation tasks: Single-domain Multilingual NMT, Multi-domain bilingual NMT, and Multilingual Multi-domain NMT. Notably, in the single-domain multilingual NMT task, CDSTX significantly enhances zero-shot domain translation performance, achieving improvements of up to +30 BLEU points through the utilization of back-translation techniques. Even in the bilingual multi-domain NMT task where specific domain data for the target translation direction is unavailable, our method consistently outperforms all SOTA methods. Moreover, in the most challenging Multilingual Multi-domain NMT task, CDSTX surpasses most baseline methods and attains a remarkable gain of +3.0 to +4.6 BLEU points.