Consistency Regularization for Translational Compositional Generalization

The ability to generate unseen combinations of known components, known as compositional generalization, is crucial in machine translation. Although neural machine translation performs well on in-distribution test sets, its performance declines when faced with out-of-distribution data, revealing limitations in handling unseen combinations. Inspired by the inductive bias which requires translation models to behave consistently when interpreting a symbol with the same meaning across different contexts, we propose consistency regularization involving representation and prediction consistency. Representation consistency encourages contextualized token representations to cluster within a distinctive representation space, while prediction consistency enhances the stability of token predictions against noise perturbations. Furthermore, to enhance the effectiveness of consistency regularization, we leverage the strong generative capabilities of large language models (LLMs) and balance the training data distribution with additional synthesized translation data. Experiments on three machine translation datasets show the effectiveness of our proposed framework. Specifically, on the CoGnition CG-test benchmark, our method demonstrates superior accuracy and consistency in translating novel phrases compared to existing approaches. It also outperforms fine-tuned LLaMA2-13B and other zero-shot LLMs, achieving state-of-the-art results. Notably, our method reduces instance-level CTER by 2.9% and aggregated-level CTER by 7.1% relative to the best-performing LLMs. Furthermore, the improvements observed on the OPUS EN $\rightarrow$ NL and IWSLT17 benchmarks further validate the efficacy of our proposed framework.