Comparison of the Psychometric Properties of Mathematical Problem-Solving Assessment Tools for Grade 7 Students: AI-Generated vs. Human-Developed

This study aims to compare the psychometric properties of mathematics problem-solving assessment tools generated by Generative AI (Chat GPT) and those developed by humans. The participants included 130 students from 4 schools, selected through stratified sampling based on school size, out of 84 schools under the secondary educational service area office Khon Kaen. The mathematical problem-solving assessments (AI-generated and human-developed) were administered via Google Forms. The study employed Construct Modeling and Unidimensional Rasch Analysis to evaluate validity evidence, reliability evidence, quality of item fit, and the Turing test. The findings indicate that both tools are similar in content validity, but the AI-generated assessment demonstrates slightly higher accuracy in several aspects. Nevertheless, the human-developed assessment offers a better balance between the test difficulty and the examinees' ability. Human content evaluation remains crucial to ensure the material adheres to appropriate assessment standards.