Zero-BertXGB: An Empirical Technique for Abstract Classification in Systematic Reviews

Abstract classification in systematic reviews (SRs) is a crucial step in evidence synthesis but is often time-consuming and labour-intensive. This study evaluates the effectiveness of various Machine Learning (ML) models and embedding techniques in automating this process. Five diverse datasets are utilized: Aceves-Martins (2021), comprising 1,258 excluded and 230 included abstracts on the utilization of animal models in depressive behaviour studies; Bannach-Brown (2016), with 896 excluded and 73 included abstracts focusing on the methodological rigour of environmental health systematic reviews; Meijboom (2021), containing 599 excluded and 32 included abstracts on the retransitioning of Etanercept in rheumatic disease patients; Menon (2022), with 896 excluded and 73 included abstracts on environmental health reviews; and a custom Clinical Review Paper Abstract (CRPA) dataset, featuring 500 excluded and 50 included abstracts. A significant research gap in abstract classification has been identified in previous literature, particularly in comparing Large Language Models (LLMs) with traditional ML and Natural Language Processing (NLP) techniques regarding scalability, adaptability, computational efficiency, and real-time application. Addressing this gap, this study employs GloVe for word embedding via matrix factorization, FastText for character n-gram representation, and Doc2Vec for capturing paragraph-level semantics. A novel Zero-BertXGB technique is introduced, integrating a transformer-based language model, zero-shot learning, and an ML classifier to enhance abstract screening and classification into “Include” or “Exclude” categories. This approach leverages contextual understanding and precision for efficient abstract processing. The Zero-BertXGB technique is compared against other prominent LLMs, including BERT, PaLM, LLaMA, GPT-3.5, and GPT-4, to validate its effectiveness. The Zero-BertXGB model achieved accuracy values of 99.3% for Aceves-Martins2021, 92.6% for Bannach-Brown2016, 85.7% for Meijboom2021, 94.1% for Menon2022, and 98.8% for CRPA. The findings indicate that the Zero-BertXGB model, alongside other LLMs, can deliver reliable results with minimal human intervention, enhancing abstract screening efficiency and potentially revolutionizing systematic review workflows.