Framework for Topic Modeling using BERT, LDA and K-Means

In today's advanced world, the demand and the amount of data being generated is increasing rapidly. The task of keeping similar data/documents together has also become tremendously difficult. Topic Modeling can help in solving this problem. It is a statistics-based NLP data mining approach used for grouping together similar documents with different contents under similar topics. We have worked on various trending topic modeling techniques and applied them on 20- Newsgroup dataset. We have worked on a novel state of art Topic modeling framework using BERT, LDA and K-means for classifying news articles under similar topics. We also performed our experimentation using simple word embedding models like Word2Vec, Doc2Vec and also by combining doc2vec, sentence transformer with bert summarizer. We were able to achieve the best NMI score out of these models by using our hybrid Bert and LDA model.