Comparison of Transformer Models for Performance on Domain Specific Texts: A Systematic Evaluation of Intrinsic Model Performance

This paper presents a systematic evaluation of transformer models for Natural Language Processing (NLP) tasks on domain-specific text. The objective of the study is to understand if intrinsic performance statistics can help deter-mine how suitable a transformer model is for a given target corpus and NLP task, where directly measuring performance is not possible because of lacking labeled data. A number of pre-trained transformer models are considered, including BERT, BART, SciBERT, RoBERTa, and XLNET which are evaluated by measuring the intrinsic performance of such models, by examining the distribution of sentence embedding vectors in the sematic hyperspace, prior to applying any down-stream NLP task. We consider this intrinsic model performance for domain-specific texts (in our case on military do-main texts as well as texts on which the models were originally trained) in the case when that text deviates from the domain on which the model was originally trained. The results of the study provide insights into the characteristics and intrinsic performance of different transformer models on domain-specific text. Our results can be used to inform the selection of a model for downstream NLP tasks given an input dataset. Overall, the study contributes to the understanding of the trade-offs between different transformer models for NLP tasks on domain-specific texts, and how to make informed decisions about which model to use for a specific input dataset. This pa-per was originally presented at the NATO Science and Tech-nology Organization Symposium (ICMCIS) organized by the Information Systems Technology (IST) Panel, IST-200 RSY — the ICMCIS, held in Skopje, North Macedonia, 16-17 May 2023.