DiFuzzNMT: A Differential Fuzzing Framework for Neural Machine Translation

Neural machine translation (NMT) systems have been widely deployed in real-world applications. However, despite the great translation performance, these systems inevitably face robustness issues and sometimes produce erroneous outputs, particularly in complex and ambiguous scenarios. In this work, we propose a Fuzzing framework with Differential Testing for NMT systems, namely, DiFuzzNMT. DiFuzzNMT employs a heuristic strategy to continuously search inputs that produce greater output differences across various NMT systems for error detection. Specifically, DiFuzzNMT establishes token mappings between outputs of different NMT systems for the same input through word alignment. To guide the test input generation process of fuzzing, we design specific testing guidance that takes into account the differences between outputs, including word alignment differences and token semantic differences. All the test inputs are generated based on “seed” inputs (inputs to generate new inputs) by applying a mutation operator. Test inputs exhibiting higher testing guidance values are selected as new seeds, while the others are discarded. A potential translation error is reported when the same test input exhibits significant differences across different NMT systems. By iteratively retaining seeds and generating test inputs, DiFuzzNMT can effectively detect translation errors. To evaluate the effectiveness of DiFuzzNMT, we conduct experiments on two widely used NMT APIs (Baidu Translate and Tencent Translate), using a publicly available dataset of 800 original sentences across 8 thematic categories. The experimental results show that DiFuzzNMT detects more translation errors than baselines and exhibits greater diversity. Furthermore, the results show that the proposed testing guidance improves the method's ability to detect translation errors.