Online Emerging Topic Detection on Twitter Using Random Forest with Stock Indicator Features

Social media is one of the most impactful and fastest communication methods. By monitoring Twitter streams, we are able to detect emerging topics and understand events around the world. There are some prior attempts that aim to online detect topics on Twitter. However, they can only detect bursty topics by using user-defined keywords a long with simple rules. In this paper, we propose an algorithm to detect emerging topics on Twitter streams. To detect emerging topics, a clustering technique has been applied to aggregate a set of keywords. Since an emerging topic occurs continuously, the emerging topics are merged with stateful technique to accumulate topics from different time intervals. To detect both high signal topics and small-medium signal topics, we use statistical features based on average, acceleration, and z-score. Moreover, we propose to include the stock indicator features: Relative Strength Index (RSI) and Stochastic Oscillator (STOCH). They are common features in trend (oversold and overbought) detection in stock analysis which is similar to our topic detection in twitter. To capture any event patterns, Random Forest (RF) has been proposed as a classifier to detect emerging keywords by utilizing the stated above five features. To evaluate the performance, we created and published a corpus by collecting Twitter data for 10 days with over 80 million tweets and then labeling possible topics in total161 events along with related keywords. The experiment was conducted on our collected data. The F1-results show that our model outperforms all baselines: TwitterMonitor, SigniTrend, and TopicSketch, in terms of detected keywords and topics.