Evaluating Medical Translation Quality Using Neural Machine Translation Models: A CUI-Based Approach via Concept Embeddings from UMLS Data

This study evaluates how effectively Neural Machine Translation (NMT) models handle medical terminology translation by introducing a new evaluation approach that combines Clinical Unique Identifiers (CUIs) and cui2vec embeddings. Using two pre-trained NMT models (Facebook's m2m100 418M and Helsinki-NLP's opus-mt-en-es), we assess their semantic accuracy in medical translations through both traditional lexical metrics and CUI-based semantic metrics. The findings highlight both capabilities and limitations of NMTs in specialized medical translation, while suggesting paths for improving their precision in critical domains.