Annotation Framework for Hate Speech Identification in Tweets: Case Study of Tweets During Kenyan Elections

Considering the colossal amount of user-generated content on social media, it has become increasingly difficult to monitor hateful content being published on public online spaces, especially during the electioneering periods, particularly in Kenya. In this regard, it is crucial to automate the identification of hate speech in order to manage the volume, variety, veracity and velocity of this content. In this research, we postulate a supervised machine learning approach whereby annotation of the training data set is critical in determining the performance of the trained classifier. Therefore, we develop an annotation framework based on Sternberg's (2003) hate theory and test its performance in classifying about 5k tweets using 3 human annotators per tweet. Preliminary results indicate an intercoder reliability score of 0.5027 based on Krippendorff's alpha.