Predicting Software Engineering Trends from Scientific Papers with a Combined Framework of Clustering and Topic Modeling

The rapid growth in scientific publications has led to the constant emergence of new trends across various fields. Predicting trends in software engineering is crucial to aid researchers and practitioners stay ahead of the curve. This paper proposes a framework that predicts emerging trends in the software engineering field by analyzing 76,689 research papers from 63 publication sources from 1980 to 2022 (reserving 2023 and 2024 for testing). The approach includes preprocessing abstracts and titles, encoding the tokens using the "allenai-specter" Sentence-BERT (SBERT) model, applying dimensionality reduction with Principle Component Analysis (PCA) followed by Uniform Manifold Approximation and Projection (UMAP) to reduce the high-dimensional data, improving cluster separation. Subsequently, clustering is performed using K-means and hierarchical clustering algorithms, with the latter providing more meaningful groupings. Latent Dirichlet Allocation (LDA) is then applied to identify topics that get labeled automatically using Chat GPT (mini o1) API. Trend analysis is conducted by examining temporal changes in topic popularity, allowing the identification of emerging and declining topics over time. The results prove that using dimensionality reduction and clustering prior topic modeling improves the generated topics; being more descriptive and informative. They also show that hierarchical clustering yields the most semantically meaningful topics, and combining abstracts with titles significantly improves topic cohesion. Trend analysis reveals growing areas such as Software Development, Application Security, and System Testing, while highlighting the decline of other outdated topics.