QDLTrans: Enhancing English Neural Machine Translation With Quantized Attention Block and Tunable Dual Learning

Neural Machine Translation (NMT) is a fundamental task in natural language processing, typically relying on large-scale parallel corpora to achieve high translation quality. However, resource-scarce language pairs, such as English-to-Vietnamese, suffer from a lack of high-quality bilingual data, which limits the effectiveness of NMT models. This paper proposes QDLTrans, a framework designed to enhance translation performance under resource-scarce conditions by integrating the multilingual pre-trained model ML-BERT into the Transformer architecture. To address the complexity of incorporating pre-trained language models, we introduce a Quantized Attention Block (QAB), which effectively distills semantic information from ML-BERT and integrates it with the Transformer. Additionally, we propose a Tunable Dual Learning (TDL) strategy that enriches the training process by leveraging both forward and reverse samples, significantly improving bilingual representation. Through a two-stage optimization process, we mitigate knowledge interference during fine-tuning using DoRA (Weight-Decomposed Low-Rank Adaptation), further enhancing model performance. Experimental results on resource-scarce translation datasets (i.e., IWSLT’14 En-De, IWSLT’15 En-Vi) demonstrate that QDLTrans achieves substantial improvements over baseline models with only a slight increase in computational costs. Specifically, our method outperforms existing approaches in both translation quality and efficiency, offering a balanced solution for enhancing NMT in low-resource settings.