A Systematic Review of Pretrained Models in Automated Essay Scoring

Automated essay scoring (AES) uses artificial intelligence and machine learning methods to grade student essays and produce human-like scores. AES research has witnessed significant advancements over time by adopting diverse machine learning models. This evolution started with traditional techniques like regression and classification, and later advanced to deep learning models that leverage neural networks for enhancing scoring performance. This review focuses on the utilization of Transformer architectures, a sophisticated form of neural networks employing attention mechanisms, with an emphasis on pretrained models like BERT in AES research. The aim is to enhance the understanding of their applicability in advancing the AES research landscape. Additionally, this study selected and analyzed relevant papers from Scopus and Web of Science databases in the past five years, by adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. An intensive screening process was followed to shortlist studies within the defined scope, focusing on AES applications that employed pretrained models. From the comprehensive analysis of 354 studies, this review shortlisted 22 key papers and identified five distinct focus areas within the current AES research landscape: holistic scoring, trait scoring, cross-domain and generalization, model fairness, and robustness. The results highlight the potential of transformer-based pretrained models in improving AES systems’ accuracy. However, pretrained models face several limitations, such as high computational costs, long input text length, and explainability. Tackling these challenges and integrating emerging technologies, such as large language models (GPT4), is expected to foster the development of accurate, robust, and transparent AES systems.