Detect Me If You Can - Towards Cross-Lingual Comparison Framework of AI Text Detection Tools Performance

In this paper, we propose a framework for evaluation of cross-lingual performance of AI text detection tools. The framework consists of the sample selection methodology and a statistical apparatus. To illustrate the use of the proposed framework, performances of five AI text detectors in English, representing a larger linguistic domain, and Serbian, representing a smaller linguistic domain, are compared. The analysis also investigated the impact of automatic translation on detectors' performance. Results indicate that, as expected, detectors perform better for English texts, but there is an instance where performance is comparable for Serbian. Additionally, a case was found where automatic translation can mislead detectors. This study should shed light on limitations and capabilities of existing detection technologies and potential directions for improvement.