Tracking Themes and Trends in a Brief News Post on Social Media: A Topic Modeling Approach

In every sector, the rapidly changing pace of change gives rise to many innovative opportunities. The proposed research investigates the novel integration of Term FrequencyInverse Document Frequency (TF-IDF) and Latent Dirichlet Allocation (LDA) to detect the underlying themes in a collection of articles. Unlike other topic modeling methods, the present research reinforces topic uniqueness by adding TF-IDF weighting to LDA, making the topics more coherent and interpretable. The optimal number of topics is decided using coherence scores, with an impressive coherence score of 0.59 at 10 topics, reflecting a better trade-off between interpretability and accuracy than in traditional methods. The study also indicates the significance of specific words, for example, “COVID,” which had the highest TFIDF value, emphasizing its occurrence in the corpus. Additionally, advanced visualization techniques like word clouds and heatmaps have been applied to enhance readability and make principal themes more readily understandable. This study far outperforms existing research in achieving better coherence scores and more understandable topic distributions, demonstrating that the TF-IDF-improved LDA model can effectively process short texts.