Examining the Accuracy of AI Detection Software Tools in Education

Educators have raised concerns about the utilization of ChatGPT in generating unoriginal text and the possibility of plagiarism. To address these concerns, various AI text detection software tools have been developed to evaluate whether a text is AI generated or human generated. The aim of this research is to empirically examine the accuracy of AI detection tools in identifying AI-generated texts. An experiment was conducted using textual data generated by ChatGPT, which was assessed using Turnitin and four other AI detection tools. Through multiple iterations and interventions, the text was paraphrased by ChatGPT until it appeared original and could not be detected as AI-generated by Turnitin’s AI detection tool. The findings revealed that all the AI detection software tools that were examined failed to detect the AI-generated text by ChatGPT in the final iteration. The findings provide valuable insights that have implications for various stakeholders, including educators, researchers, and AI text detection software developers. Based on the tools examined, educators and researchers do not need to set a specific threshold or percentage, including 0%, to determine what qualifies as acceptable AI-generated text. This is because establishing such a threshold can be misleading, considering the current limitations in the algorithms of these tools. Furthermore, the data generated in this paper can provide a solid basis for replicated research or software testing and assessment. It can be utilized to evaluate the accuracy of alternative AI detection tools and any future advancements in the tools mentioned in this investigation.