Evaluation of Large Tweet Dataset for Emotion Detection Model: A Comparative Study between Various ML and Transformer

Specific emotion detection in written human language is a challenging problem in various research fields, including psychology, neuroscience, and computer science. Twitter is a suitable source for collecting large emotion datasets, as users have provided tweets with emotion hashtags (e.g., #fear, #anger, #sadness, #joy, #surprise, and #disgust) expressing their emotions. However, the criteria for data collection, i.e., the position of representative or synonymous emotion hashtags, remains unclear. Next to this unclarity, we assess the suitability of various machine learning (ML) algorithms for this purpose. In this study, we collected over five million tweets (n=5,645,139) with 24 emotion hashtags and investigated the efficacy of different criteria for collecting tweets. Contrary to previous research, we found that applying any position of representative emotion hashtags can achieve strong performance, rather than applying the last position of synonymous emotion hashtags. Our study shows that the RoBERTa-large transformer model outperforms deep learning algorithms and traditional ML algorithms in terms of specific emotion detection in tweets, especially when trained on a dataset with a balance between size and quality. We also found that larger datasets are more efficient for RoBERTa model training than smaller datasets. Along with these empirical contributions, we share the collected emotion dataset.