A Novel Integration of Hierarchical Clustering and Ensemble Classification Algorithms for News Classification

This research work proposes a novel approach for news classification by integrating hierarchical clustering and ensemble algorithms. The proposed methodology began with the acquisition and preprocessing of the BBC News dataset taken from Kaggle. Text data underwent preprocessing by including tokenization, removal of stop words, and TF-IDF vectorization. Hierarchical clustering was then applied to group similar documents based on their content by using the Agglomerative Clustering algorithm. The number of clusters was adjustable for varying granularity. Subsequently, cluster labels obtained were assigned to corresponding documents, and additional features were extracted from the clustered data. These features, alongside the TF-IDF matrix, served as an input for training Gradient Boosting, Bagging Classifier and Random Forest classifiers. These classifiers were trained to predict document categories based on both text features and cluster assignments. The trained models were then tested on a separate test set to evaluate their generalization ability. Clustering and classification results were carefully evaluated to assess method efficacy, elucidating the relationship between document clustering and classification performance.