A Pilot Study on the Development and Validation of AI Literacy Test Items for Grade 7 to Grade 9 Students

To provide K-12 students with an understanding of AI’s potential opportunities and challenges, AI education has been incorporated into the technology curriculum. Most existing methods for assessing students’ AI literacy rely on self-assessment questionnaires, which may not offer an objective measure of their understanding. This study, therefore, sought to develop and validate a 25 -item multiple-choice test tailored for Hong Kong secondary school students to gauge their AI literacy. A total of 144 students from six secondary schools participated in a pilot test. Item Response Theory (IRT) incorporating a Markov Chain Monte Carlo (MCMC) algorithm, was developed to estimate the difficulty and discrimination indices of test items. The 3-Parameter Logistic (3-PL) model was selected for its effectiveness in providing a more nuanced understanding of item characteristics. Preliminary analysis suggested that the test warrants further revision and enhancement, as evidenced by a Kuder-Richardson Formula 20 (KR-20) reliability coefficient of 0.68. Nonetheless, all test items exhibited satisfactory discrimination capabilities, indicating their proficiency in differentiating among varying levels of respondent aptitude. Thus, this study can provide a foundation for developing a standardized AI literacy test in the future.