Is ChatGPT Reliable in Scoring Learner's Translation Quality?

In order to investigate the application of large language models in foreign language teaching and learning, we employed ChatGPT for grading students' translations. We studied the reliability of ChatGPT evaluation of Chinese-to-English translations on five topics in real-world setting. This study conducted the analysis of impact of prompt crafting which guides ChatGPT to generate response, compared the different performances with and without reference in scoring, and tested the ability of ChatGPT on cross-lingual and similarity comparison. Experimental results reveal that correlation of the scores assigned by ChatGPT with those marked by human raters is rather low. The scores generated by ChatGPT are fluctuant with different time, prompts and topics. Furthermore, these generated scores tend to be neutral and are not sufficiently differentiated among translations of different qualities. The study presents a critical view of the application of ChatGPT to automatic learner's translation scoring task.