News in Vectors: Optimizing News Analysis through Advanced Embeddings and Clustering

With the advent of mass media and news dissemination methods such as newspapers, magazines, online articles, etc there has been an increase in data. However, this omnipresence of information and data poses a new challenge to organisations-that of organising and recognising various categories of data. Manual organisation may seem straightforward but it can cost a lot of resources and manpower that could be effectively used for better purposes. This is where topic modeling and clustering of news data plays an important role. Clustering is an unsupervised learning algorithm where an unlabelled dataset is used and clustered into several categories based on their similarity with each other. In this study we have experimented with various clustering algorithms- kmeans, k-mediods, hierarchical (agglomerative) and DBSCAN, along with two vectorization techniquesDoc2Vec and sentence transformer. This work leverages the BBC News Articles dataset for this purpose. Eight different pipelines have been generated with the aforementioned clustering and vectorization techniques and the result of the best combination has been depicted. The best performing pipeline gives a silhouette score of 0.56. The outcomes of the pipelines have been compiled, compared and analyzed by visual representation.