Semantic Structure Invariance-Based Metamorphic Testing for Machine Translation Systems

In recent years, deep neural networks have been applied in machine translation systems, resulting in the so-called neural machine translation (NMT) models that can improve translation quality significantly. However, due to the brittleness of deep neural network, machine translation systems could return erroneous translations that lead to misunderstandings or even cause serious losses. To detect translation errors, various testing techniques have been proposed. As a popularly used technique, metamorphic testing mainly relies on text or syntactic structure of translations while ignoring the meaning of sentences (i.e., semantic information). Compared with text and syntactic information, semantic information of sentences is more stable when dealing with languages that have rich vocabulary and flexible word order. Motivated by this observation, we propose semantic structure invariance-based metamorphic testing (SSIMT) for machine translation systems. The key insight is that contextually similar sentences should typically have translations of similar semantic structures. Experiments have been conducted to evaluate SSIMT on two widely used machine translation systems, Microsoft Bing Translator and Google Translate with 600 seed sentences crawled from well-known news websites covering six different corpus topics. The experimental results show that SSIMT is able to find thousands of erroneous translations in both translation systems with high accuracy (over 70%). Translation errors reported by SSIMT covers a wide variety of common error types.