LDA based semi-supervised learning from streaming short text

With the rapidly growing of real-time social media, like Twitter, many users share and discuss their interest topics through such platforms. Hashtag is a type of metadata tag which allows users to annotate their topics of tweets. For research usage, for example, hashtags can help the performance of event detection by observing the trend of hashtags. Although Twitter grows rapidly, hashtag growth is not as expected. Our dataset shows that there are less than 20% of all tweets containing hashtags. We think that it is caused by that most users may have no idea what hashtags are suitable for tweets they post. If we can recommend suitable hashtags to users, it can be one of the solutions to solve the problem of low usage rate of hashtag. Hashtag recommendation belongs to supervised learning problem. More labeled data for training the learning model can get higher performance in prediction. However, labeled data in hashtag recommendation is not so much due to low usage rate of hashtag. Thus, we want to exploit unlabeled data, i.e. non-hashtag tweets, to solve this problem. Now we have large amount of unlabeled data, but directly adding all non-hashtag tweets may not be helpful to train the model. To overcome this issue, we apply the weight-updating mechanisms to filter out the useless parts of non-hashtag tweets. These mechanisms also have to consider the temporal characteristics of hashtag due to the real-time nature of Twitter. The experimental results in this research show that adding non-hashtag tweets to extend original training data outperforms baseline methods which only exploit labeled data to train the model.