An Analysis of Research Trends on Language Model Using BERTopic

Although language models have played a crucial role in various natural language processing tasks, there has been little research that focuses on systematic analysis and review of research topic trends in these models. In this paper, we conducted a comprehensive analysis of 31 years of research trends in the field of language models, using publications from Scopus, an internationally renowned academic database, to identify research topics related to language models. We adopted BERTopic, a state-of-the-art topic modeling technique, on the 13,754 research articles about language models. The research on language models has gradually increased since 1991, and there is a sudden increase in the number of publications with the emergence of BERT and GPT in 2018. We assigned 14 main topics with meaningful keywords clustered by BERTopic model. Among 14 topics, research related to speech recognition, statistical language models, and pre-trained language models demonstrated the most vigorous research fields. Our results demonstrate a more systematic and comprehensive trend in language model research, which is expected to provide an important foundation for future research directions.