Evaluating Psychometric Properties in Generated Thai Reading Ability Diagnostic Tests between the Large Language Model (LLM-Claude) and Human: Rasch Analysis

This study aims to compare the quality assessment of Thai reading comprehension diagnostic tests created by Claude AI versus those developed by humans. The sample consisted of 735 seventh-grade students from Secondary educational service areas in the central northeastern region of Thailand. The methodology applied Rasch Model Analysis integrated with Turing Test procedures. The findings revealed that diagnostic tests created by both Claude AI and humans demonstrated comparable measurement quality in terms of validity, reliability, and item-model fit. Both tests exhibited low measurement error, allowing for accurate estimation of students' Thai reading abilities close to their true proficiency levels. Furthermore, both test versions showed good distribution of difficulty levels, covering nearly the full spectrum of student ability levels. These characteristics make the tests particularly suitable for students with slightly above-average proficiency. Nevertheless, certain test items require refinement to enhance their assessment efficiency according to established criteria. While some educators may remain hesitant about implementing AI-generated tests for formal student evaluation, Claude AI-created tests can effectively serve as practice exercises for student development.