Hot Topic Detection on Twitter Data Streams with Incremental Clustering Using Named Entities and Central Centroids

Nowadays, hot topic detection is an important knowledge discovery task on social data streams to determine hot topics that are being discussed the most on a social platform. One of the main challenges of this task is the processing of a very large social dataset in a sequence order efficiently and effectively. In this paper, our work proposes a novel incremental clustering-based solution with named entities and central centroids for hot topic detection on Twitter data streams. Using named entities, each tweet and its cluster can be semantically represented. They are then efficiently used for searching in noise removal and incremental clustering. After that, clusters of higher quality can be formed and lay the basis for deriving true hot topics. Besides, central centroids are defined instead of normal centroids to speed up our incremental clustering process. As a result, our solution is more efficient and effective than the other approaches in the existing works as shown in an empirical evaluation on Twitter Events2012 dataset. Indeed, our entity and central centroid-based incremental clustering method outperforms the others with Recall of about 0.92, Normalized Mutual Information of about 0.85, and execution time of about 30 minutes on processing 500,000 tweets with more than 500 topics. Such better results confirm the appropriateness of our new method design features for hot topic detection on Twitter data streams.