Approach to Evaluating AI-Generated Educational Content: A Case Study on AI-Generated Tests

Evaluating texts generated by large language models (LLMs) remains a critical challenge due to their high linguistic quality, susceptibility to hallucinations, and potential lack of depth. Existing evaluation methods, including lexical analysis, hallucination detection, and LLM-based assessments, often fail to comprehensively assess the substantive meaning of texts in educational contexts. This study focuses on the evaluation of LLM-generated educational materials, particularly test questions, identifying key limitations in conventional assessment approaches. To address these challenges, we propose a novel evaluation framework based on knowledge graphs automatically constructed by LLMs using contextual information from lecture materials and course syllabi. This graph-based approach, aligned with Bloom’s taxonomy, enables a structured and automated assessment of AI-generated educational content, improving its reliability and pedagogical validity. While the primary goal of this method is to evaluate the quality of educational materials, it also has the potential to be applied in the future for their refinement and improvement, ensuring better alignment with learning objectives.