StructureTester: Automatic Machine Translation Testing Based on Variation Feature Vector

In recent years, the performance of machine translation systems has made remarkable progress, primarily due to the rapid advancements in neural network language models. These state-of-the-art models enable the swift translation of vast amounts of text, leading to considerable time and cost savings. In pursuit of enhancing machine translation accuracy, researchers have devoted attention to developing automated translation testing tools. A prominent approach in this context involves comparing the translation results of “similar” source sentences, anticipating the correctness of translation by similarities in sentence structure. However, despite the potential of this approach, the current studies still face certain challenges. Notably, false negatives and false positives persist as issues. Moreover, achieving high detection accuracy for all types of translation errors remains an ongoing challenge. To address these challenges, we propose the StructureTester, a novel approach that not only leverages the differences between the structure trees of two sentences but also employs changes in sentence purpose as crucial judgmental features. Our proposed method yields significant improvements, elevating the overall detection accuracy to an impressive 98.17%. Furthermore, StructureTester effectively identifies various types of translation errors.