A Word Embeddings and Stylistic Features based Approach for Generative AI Authorship Verification

The remarkable capacity of generative large language models such as GPTs, to generate high-quality content in a variety of fields that nearly mimics human writing has garnered attention in recent years, posing significant problems in differentiation. Consequently, it is now more important than ever to detect machine-generated text. PAN 2024 organizes a competition for differentiating the text written by human or machine. They provided the dataset of human writings and machine generated texts. In this work, we developed a hybrid model for predicting whether the text was generated by the human or a machine. The proposed model integrates the stylistic features and embeddings from two sophisticated transformer models, such as RoBERTa and T5 models. The PAN 2024 Generative AI Authorship Verification dataset is used in this experiment. The dataset contains files of concepts that are written by humans and different large language models. In the proposed model, the files are represented as vectors by combining the stylistic features based vector representation and word embedding techniques based vector representation. These vectors are forwarded to LightGBM algorithm for training. The performance of proposed model represented in terms of numerous evaluation metrics such as ROC-AUC, C@1, Brier, F0.5u, F1, and mean. The proposed hybrid model attained scores of 0.996 for ROC-AUC, 0.970 for Brier, 0.989 for C@1, 0.972 for F1, 0.971 for F0.5u, and 0.979 for mean on the training dataset. These results demonstrate the importance of including transformer embeddings along with stylistic features to improve the performance of authorship verification.