Unsupervised Analysis of Scientific Paper Abstracts: Exploring Clustering Algorithms

The number of scientific publications published online has increased significantly in tandem with the rise of the Internet. The quantity of online articles has significantly expanded in recent years. As the quantity of scientific papers continues to grow exponentially, researchers are having challenges organising and generating relevant insights from the vast amount of information accessible. Unsupervised clustering techniques are applied in this research to categorise scientific papers based on the abstract. Centroid Based Clustering, Spectral Clustering, Distribution Based Clustering (DBSCAN), Grid Based Clustering, and hybrid LDA-BERT KMeans Clustering are the clustering algorithms explored in this study. Results showed that Centroid-Based Clustering resulted in 15 clusters with a silhouette score of 0.0038, Spectral Clustering produced 10 clusters with a silhouette score of 0.0046, DBSCAN formed a single cluster only, Grid-Based Clustering achieved 82 clusters with a silhouette score of 0.3061, and Hybrid LDA-BERT KMeans Clustering yielded 6 clusters with a silhouette score of 0.3951. Possible reasons that the algorithms are less effective compared to one another in this specific task are also investigated.