AI-Based Examination Content Creation: Evaluating Large Language Models for Question Generation

Automated multiple-choice question (MCQ) generation has the potential to revolutionize AI-driven educational assessments. This study evaluates the performance of three state-of-the-art large language models—LLaMA 3.2, Mistral 7B, and Qwen 2.5-Max—across three benchmark datasets: SciQ for scientific reasoning, RACE for reading comprehension, and Cosmos QA for commonsense reasoning. Fine-tuned using Low-Rank Adaptation (LoRA) and 4-bit quantization, the models were assessed using F1-score, ROUGE, BERTScore, and Token Effi-ciency to quantify accuracy, contextual alignment, and efficiency. Experimental results show that Mistral 7B achieved the highest accuracy on SciQ, with an F1-score of 0.891, demonstrating strong factual consistency. Qwen 2.5-Max excelled in RACE (F1 = 0.8252) and Cosmos QA (F1 = 0.702), outperforming in multi-step reasoning and implicit inference. LLaMA 3.2, while effective in structured MCQs, showed limitations in handling implicit dependencies, achieving its lowest score on Cosmos QA (F1 = 0.646). ROUGE-L analysis further highlights Mistral 7B’s dominance in structured scientific assessments (0.3699) and Qwen 2.5-Max’s superior contextual coherence in reasoning-based tasks (0.3209). Additionally, BERTScore confirmed Qwen 2.5-Max’s strong semantic alignment (0.86), while Mistral 7B balanced efficiency and factual precision. These findings underscore the importance of selecting models based on the complexity of assessment tasks. While Mistral 7B is best suited for structured, fact-based question generation, Qwen 2.5-Max excels in inference-heavy and multi-step reasoning tasks. Future improvements should focus on mitigating bias, enhancing dataset diversity, and refining domain-specific fine-tuning techniques to further advance AI-driven question generation in educational applications.