Ensemble Machine Translation to Filter Low Quality Corpus

The popularization of ubiquitous computing equipment and the development of network communication technology have greatly improved the productivity of human language information. The quality of large-scale language information is a key factor affecting the effectiveness of deep learning algorithms. How to filter low quality corpus from large-scale corpus that is unstructured, non-standardized, and even contains fallacies, to obtain high quality corpus has become a research issue of great application value. We focus on the specific filtering issue of bilingual parallel corpus, re-examine the manual filtering process, and propose an ensemble machine translation filtering idea. From the perspective of translation direction, we design and implement a single-engine-based ensemble machine translation filtering framework and algorithm, and from the translation system perspective, we design and implement a multi-engine-based framework and algorithm respectively. Experimental results show that the single-engine-based algorithm can integrate the advantages of different machine translation directions, and the multi-engine-based one can integrate the advantages of different machine translation systems. Using the straightforward methods of Levenshtein string morphological similarity and linear weighted ensemble can implement efficient industrial-grade corpus filtering.