Hateful Sentiment Detection in Real-Time Tweets: An LSTM-Based Comparative Approach

It is undeniable that social media has improved our lives in many ways, like allowing interactions with others all over the world and network expansion for businesses. However, there are detrimental effects of such accessibility, including the rapid spread of hate through offensive messages typically directed toward gender, religion, race, and disability, which can cause psychological harm. To address this problem of social media, many researchers have recently proposed various algorithms powered by machine learning (ML) and deep learning for the detection of hate speech. This work proposes a hate speech detection model based on long-short term memory (LSTM), using term frequency inverse document frequency (TF-IDF) vectorization, and makes comparisons with support vector machine (SVM), Naïve Bayes (NB), logistic regression (LR), XGBoost (XGB), random forest (RF), <inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula>-nearest neighbor (<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula>-NN), artificial neural network (ANN), and bidirectional encoder representations from transformers (BERT) models. To validate and authenticate our proposed work, we obtained and classified a real-time Twitter data stream of a trending topic using Twitter API into two classes: hate speech and nonhate speech. The precision, recall, and <inline-formula> <tex-math notation="LaTeX">$F1$ </tex-math></inline-formula> score achieved by LSTM are 0.98, 0.99, and 0.98, respectively. The accuracy of LSTM for detecting hateful sentiment was found to be 97%, surpassing the accuracy of other models.

meaning that people have the freedom to express anything they want, whether it is positive or negative.Recently, the number of hate posts on topics, such as politics, racism, nationality, and religion, has largely increased, especially during the COVID-19 lockdown.The manual identification and removal process of such posts is a cumbersome job considering the seemingly infinite number of hateful tweets.Hence, classifying and detecting hateful speech using natural language processing (NLP) and machine learning (ML) models are significant.Yet, it is still very difficult to perform these operations for real-time tweets on a day-to-day basis [5], [6].Therefore, we extracted a real-time stream from the Twitter platform using Twitter's application programming interface (API) in this work.In the literature, it has been found that the detection of hate speech is mostly achieved through binary classifications, multiclass classifications, or a combination of both.ML algorithms are commonly used in detection methods for hate speech, which are typically only available in the English language.In this work, we first created an English corpus containing hate and nonhate speech from a real-time stream of tweets via API that is available on Twitter.This application program is used to avail a continuous stream of tweets and help us avoid dependence on the already existing corpora of tweets available on the web.Besides, our constructed corpora of tweets add validity and authenticity to the proposed work.To deal with the corpora of tweets, such as preprocessing, normalization, and other preprocessing-related steps, we used Panda's library.Since the corpus cannot address all issues related to hate speech and its detection, we further classified the tweets using our proposed ML and deep learning models.To identify the words that are responsible for hate speech detection, we incorporated the term frequency inverse document frequency (TF-IDF) technique.This technique adds weight to the word or term to determine its significance in the whole corpus.The value of TF-IDF is proportionate to the number of times a particular word appears in a document.Considering that 83% of document searches on the Internet are conducted using TF-IDF, it seemed to be the best technique to identify hateful and nonhateful tweets in this work.
The word cloud technique was employed to further prepare a visual representation of the tweets.Moreover, we developed a comprehensive strategy to perform the challenging task of finding sentiments based on the considered hashtags using TextBlob, which is a Python-based NLP-supported framework.TextBlob returns the polarity of a sentence along with subjectivity, where a polarity of −1 refers to negative sentiment and +1 to positive sentiment.
To analyze any hashtag, online content, or anybody's Twitter account, several analytical tools are available but are not easy to develop [7], [8].Since meaningful information can be found from real-time tweets, a live extraction of tweets was one of the motivations of this work.Also, incorporating state-of-theart Python packages, such as TextBlob library, and word cloud representation of significant terms have been successfully included in this research.Moreover, utilizing different deep learning models, such as long-short term memory (LSTM) and other advanced ML, for hate speech detection was another motivational factor for this study.
Overall, the following steps were performed.1) To validate and authenticate our proposed method, we extracted real-time tweets using Tweepy API.Then, we cleaned and preprocessed tweets to create a data frame for subsequent analysis.2) We analyzed the sentiment of the tweets using the TextBlob library of Python in order to find the subjectivity and polarity scores.Based on the calculated polarity scores, the tweets were identified as hate speech or the other of two as nonhate speech.
3) The TF-IDF of the Twitter data was calculated, and n-gram feature extraction was performed to determine the importance of a particular word in the dataset.
To mark the words with the greatest contribution to the hateful sentiments, word cloud, also called tag cloud, was used to prepare a visual frame.4) The proposed LSTM model achieved an accuracy of 97% for detecting hate speech and, thus, appears to be a good choice for evaluating textual data that is sequential by nature.Besides, the comparative analysis to Naïve Bayes (NB), logistic regression (LR), support vector machine (SVM), XGBoost (XGB), random forest (RF), K -nearest neighbor (k-NN), and artificial neural network (ANN) and bidirectional encoder representations from transformers (BERT) models further confirms the effectiveness of our method.The rest of this article is structured as follows.Section II describes the related works, Section III introduces the proposed model, Section IV discusses the experimental results, Section V presents the discussion, and Section VI concludes the findings and suggests directions for future works.

II. RELATED WORKS
Many researchers have implemented different techniques based on dictionary, bag-of-word, n-gram, TF-IDF, and deep learning, for hate speech detection.For instance, Mohapatra et al. [9] investigated hate speech in a Facebook dataset containing 27 422 posts from different popular pages in mixed Odia-English language and classified the posts into three categories: "hate," "offensive," and "nonhate" speech using wordto-vector, TF-IDF, and n-gram techniques.The researchers applied SVM, NB, and RF ML models to check the features of "hate," "nonhate," and "offensive" language and found that their proposed RF model achieved the best accuracy of 75%.In another work, Bhora et al. [10] employed word-n-gram, lexicon, character-n-gram, and negation for feature extraction to evaluate an English-Hindi mixed social media text for hate speech detection and SVM as a predictive model, achieving an accuracy of 71.7%.Another work [11] focused on both topics and hate speech targets at a coarser level and discussed how that coarser level knowledge transfer happens across a variety of topics.This article also implemented a multitask approach of ML models.Sharma et al. [12] proposed a model that concentrates on examining hate speech in Hindi-English code-switched language and achieved 15% better performance compared to other existing models.In [13], the zero-shot style-transfer method is introduced, which uses an unsupervised technique for hate to nonhate conversion.García-Díaz et al. [14] analyzed which features are significant for selecting hate speech in Spanish, and a combination of linguistic features and transformers influenced their model.In [15], a deep learning-inspired multitask model that uses shared-features and task-specific features was developed, which achieved an encouraging F1 value in hate-speech detection.Moreover, Mossie and Wang [16] present a model to classify hate speech and nonhate speech in the Amharic language using Apache, an open-source tool.Table I displays some recent works on hate speech detection in social networks, including the models used, feature extraction strategy, and accuracy.

III. PROPOSED MODEL
The proposed model for detecting hate speech from realtime tweet data is shown in Fig. 1.As seen, data from a raw tweets stream are extracted using API and preprocessed, which involves removing punctuation, vectorization, and cleaning.Subsequently, using the TF-IDF technique [25], features are extracted from the collected tweets, and then, the n-gram method is used.After completing the feature extraction, the model is trained by the proposed ML LSTM algorithm.LSTM is an advanced structure of recurrent neural networks (RNNs) that is primarily helpful for sequence learning, such as NLPrelated problems.Although RNNs are not that successful in the vanishing gradient problem for analyzing tweets in terms of their sentiment, LSTM has shown to attain better accuracy than convolution networks and, thus, has found popularity among researchers.The hate speech detection problem is similar to a sequence learning problem, for which LSTM networks are a state-of-the-art method and can learn the sequence of the tweets [26].The prime reason for using LSTM is that it can learn long-term dependencies unlike RNNs [27].In this work, the results obtained by LSTM are compared with those of the SVM, NB, LR, RF, k-NN ANN, and BERT models.

A. Corpus Building of Tweets
We performed the following steps to successfully build a corpus of tweets consisting of real data streaming from Twitter.First, APIs were considered for the extraction of real-time Tweets.
Two different kinds of API are used for extracting tweets from Twitter: one for searching and the other for streaming tweets.In this work, REST API and queries were used to gather tweets from users.Based on the search of typical hashtags or terms, searching API returns the tweets that match the placed query, whereas streaming API helps to avail a continuous stream of real-time tweets from Twitter.Therefore, REST API tweepy was used for the search in this work.Twitter provides a consumer key and consumer secret key for the purpose of user authentication.On the other hand, API offers an access token and access token secret key for accessing the data.By using Oauth Handler and API authentication, the user gets authenticated, and then after using the user_time line function, any tweet can be fetched by providing the screen_name, count, lang, and tweet_mode.

B. Feature Extraction Model
For cleaning and feature extraction, we implemented the TF-IDF vectorizer, which applies the term referred to as t in document d.This TF-IDF vectorizer technique is used to find the significance of the terms in the corpus [28] where C t,d is the total number of times t appears in document d; and k C t,d represents the total number of terms that are available in document d.The significance of t in document d depends on how many times t occurs in document d.
The calculation of IDF is as follows: where D represents the total documents available in the corpus; and d t denotes the total documents that consist of the term t.The greater the value of t, the more significantly it identifies document d in the corpus of tweets.Usually, TF-IDF is computed by multiplying ( 1) and ( 2) The algorithm for finding TF-IDF is given in the following [16].The representation of text data in terms of graphical means plays a vital role in understanding the overall significance of words contributing to hateful sentiments.In this work, the word cloud technique displays significant words related to hateful comments [29].The word cloud, also known as tag cloud, usually consists of tags that are single words and shows the significance of those words using the increased font size.This form of visualization helps to understand the dominant terms or words in the tweets contributing heavily to hateful sentiments [30].

D. N-Gram
In the literature, it can be seen that text-to-feature vector conversion can be done in many ways.The most effective feature engineering methods for hate speech detection are character n-gram and the word or bag of word method.Moreover, n-gram is a contiguous sequence of n-items for a given sample of text or speech.Herein, the features extracted from the realtime tweets using Twitter API with the values of n_gram = 1 or 2 are passed in the function named generate_n-grams(), here, n refers to the number of words used in the probability

E. Resolution
The proposed ML model, LSTM, was applied to the training dataset collected from the real-time tweets using tweepy API [31].The results of LSTM were then compared with other learning algorithms, such as SVM [32], NB [33], XGB [34], RF [35], k-NN [36], LR [37], ANN [31], and BERT [38], [39].We used a supervised learning approach for detecting hate speech from the collected Twitter dataset.The models were applied to the training and test data, and the outcomes are shown in Fig. 2.
1) Support Vector Machine: SVM, originally developed by Cortes and Vapnik [32], was applied in this work to differentiate between hate speech and nonhate speech in tweets.The training vector D of SVM can be calculated using the following equation, where the n dimension of x ∈ R n signifies every instance of a tweet that is classified as either hateful or not hateful: In SVM, the differentiator is a hyperplane, which is given as where w is the weight to differentiate the hyperplane; and bias is represented as k∈ R. Furthermore, the following equations are used to show whether the hyperplane should belong to the hate speech or nonhate speech class [20]: After considering the slack variable ξ i and adjusting the misclassification rate, the above equations can be combined as 2) Bayes Classifier: In the Bayes classifier [33], we assume that a pair (X, Y ) takes a value in the range of R d X {1, 2, 3, . . ., n}, where X is considered as tuples and Y Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
is considered as a class label.The condition distribution of X provided by Y yields the value r as (X|Y = r ) ∼ P r , where r = 1 − n.The operator ∼ signifies the distribution, and P r refers to the probability of distribution.The equation for the Bayes classifier is given as The classifier C is a function R d → {1, 2, 3, . . ., n} that classifies X to the corresponding class.In this work, the class label is either hate speech or nonhate speech.
3) XGBoost: XGB is a distributed gradient boosting library that performs efficiently for text classification [34].It applies the Newton-Raphson method, which uses the Taylor approximation of second order in the loss function.The log loss function for the evaluation of classification is given as The working principle follows an ensemble learning inspired by decision trees that applies an optimized gradient boosted framework.The method tends to perform best for small to medium structured data but not unstructured data.
4) Random Forest: In RF, t number of trees are created using a discrimination function.The probability that class x belongs to class c (c = 1-2) is written as [35] The discriminant function is given as RF tends to grow as the n number of classification trees increases; subsequently, the input vector passing through each such tree in the forest also grows.The classification results are generated by each tree's "vote" for a particular test instance toward a specific class; then, the forest chooses the class with the most votes for that particular instance.

5) Logistic Regression:
The LR model is written as [37] log p(x) where the linear function p(x) is unbounded; and the value of p is between 0 and 1. 6) K-Nearest Neighbor: Using the k-NN search [40], [41], [42], the closest k neighbors from the real-time queries, i.e., tweets, are chosen for training documents.The KNN algorithm believes that similar points exist in nearby proximity, meaning similar points are nearby.Using this strategy, the algorithm predicts any unknown tweets regardless of whether they are considered hate speech or nonhate speech.For k observations near a test observation x 0 , the conditional probability that x 0 belongs to class j is determined by where k is the number of neighbors to be considered by k-neighbors queries, which was 5 in this work.

7) Artificial Neural Network:
The output function of ANN is expressed as [31] f where W = [w 1 w 2 w 3 , . . ., w n ], X = [x 1 x 2 x 3 , . . ., x n ], b is the bias, and x is the input variable, i.e., the input to the neurons.Back-propagation progressively calculates the error of each layer l; then, the δ (l) term at a particular layer l can be referred by in the following equation: The gradient can be calculated by 16) where W and b are the parameters; and x and y represent the input tweets.The back-propagation algorithm optimizes the results, or reduces the error, through propagation and weight updates.

8) BERT:
We also considered the BERT language model for the detection of hate speech as it has shown high performance in many NLP-related problems [38].Table II and Figs. 8 and 9 present the results of BERT in-terms of F1 score, accuracy, precision, and recall.While BERT [38] was developed by Google Artificial Intelligence (AI) and heavily used to solve NLP-related problems, it is now used to generate a new representation of words based on words presented in the existing tweet corpus by considering both previous and following words in the tweet.This model can check the context of a word and its relation to other words in the sequence of text via an attention mechanism.Using three embedding layers, BERT creates the final model.Three distinct types of embeddings, which can be classified as token, segment, and position, are the input to the encoder layer of the proposed BERT.In terms of a formula, it can be represented as follows: 9) LSTM Model: Finally, we considered LSTM as a prime learning algorithm to efficiently differentiate hate speech from nonhate speech [19].By incorporating three gates, including input-output, and forget gates, LSTM performs better in comparison with the other classifiers.The equations of LSTM are [26], [27] e t = σ h (W e x t + P e l t−1 + a e ) (19) j t = σ h W j x t + P j l t−1 + a j (20) Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.where t refers to time, ⃗

TABLE II IMAGE OF THE DATA FRAME AFTER SENTIMENT ANALYSIS
x t is the real number in m dimensional space, e t is the activation vector refers to forget gate, j t is the activation vector refers input gate, p t refers to the vector of activation of the output gate, L t = activation vector of hidden state, d t refers to the activation vector of the input cell, d t refers to the vector of cell state, W is the weight matrix, a refers to the bias vector, and operator o is known as the Hadamard product; and initially, d 0 = 0 and l 0 = 0.

IV. EXPERIMENTAL RESULTS
To evaluate the detection accuracy of hate speech by the deep learning-inspired LSTM and other ML models, several metrics related to performance evaluation were calculated, such as precision, recall, accuracy, and F1-Score.To perform the analysis, features were initially extracted from the dataset of tweets in order to preprocess the data.Fig. 3 displays the length of the tweets in terms of characters on the x-axis and the frequency of each tweet on the y-axis.The plot function of the matplotlib library was used to prepare the histogram and determine the output.Fig. 4 presents the subjectivity and tweets' polarity using TextBlob of Python.Fig. 5 displays the tweets and their corresponding sentiment scores.A visualization of word cloud, also called tag cloud, is illustrated in Fig. 6, which reveals the most significant word or terms that contribute to the sentiments.

A. Dataset Description
To build the dataset for this study, we used tweepy API to fetch data from a real-time stream of Twitter, specifically 150-200 tweets.By searching "Bill Gates" as the subject, we obtained 200 raw tweets.Then, using the data frame functionality of pandas in Python, we created the data frame for further preprocessing and data cleaning.

B. Preprocessing
During the data processing and cleaning, some special characters and emojis were deleted from every tweet.
Fig. 4(a) and (b) presents the line charts of the subjectivity and polarity of the extracted tweets, respectively, based on the frequency of the considered tweet.As mentioned, the subjectivity and polarity were calculated using the module TextBlob and then graphed using the plot function of the module matplotlib.Fig. 4(c) displays the labels of the extracted tweets in terms of their frequency.In addition, comprehensive sentiment analysis was performed via the TextBlob module in Python.Fig. 5 shows each tweet's assigned subjectivity, polarity, and total analysis score.Specifically, a tweet with a polarity score less than 0, which indicates a "hated" post, is assigned a value of 0. A tweet with a polarity score greater than or equal to 0, which refers to a "nonhated" post, is assigned a value of 1.

C. Feature-Extraction Results
This work applied three methods for feature extraction: word cloud, n-gram, and TF-IDF.For word cloud, we cleaned the data and obtained a visualization frame using the "Word-Cloud" module.
Afterward, we performed the TF-IDF extraction with the help of the text feature extraction module of scikit-learn provided by Python.To create the target array corpus, we achieved different scores for the vectors.Table S.1 in Appendix B contains the obtained Id and scores after applying TF-IDF.

D. Model Evaluation Results
According to the literature, the evaluation metrics for the numerous ML and deep learning techniques are typically the same for all works, which include accuracy, precision, recall, and F1 score.For example, Sadiq et al. [19] obtained 92% accuracy applying the TF-IDF-based unigram and bigram approach.Alammary [21] achieved 78% accuracy in hate speech detection from their developed approach.In this work, we employed all considered models to train the learning algorithms to evaluate the extracted real-time data from a continuous Twitter stream as hate speech or nonhate speech.By feeding the training tweets into the models, sentiment analysis was conducted on the extracted real-time tweets based on accuracy area under curve (AUC) score, and F1 score, as shown in Figs. 8 and 9, and Table II, respectively.In addition, the following equations were used to calculate for accuracy, precision, recall, and F1 score: For each model, the complete classification reports of all classifiers on test data are provided in Table II.The training accuracy for ten Epochs of LSTM was also very satisfactory, as indicated in Fig. 10.
V. DISCUSSION This study reveals the importance of the detection of hate speech on social media, particularly on the Twitter platform, which has been investigated in several works.instance, et al. [9] developed models for English-Odia mixed comments from many Facebook pages and processed them by and cleaning out hateful speech.They used NB, SVM, and RF binary models with feature extractions via word2vec, n-gram, unigram, bigram, trigram TF-IDF, and TF-IDF+ combined n-grams.Among these, SVM achieved the highest average accuracy of 72.54%.For combined n-grams, NB achieved 74.77%, and word2vec features RF obtained 75.39% accuracy.Sadiq et al. [19] proposed CNN-LSTM and CNN-BiLSTM for aggression detection and obtained an overall accuracy of 92% but failed Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.to perform a detailed analysis of the aggression detection of various features.In addition, Araque and Iglesias [20] reported the Affective Space and SenticNet-based ensemble method with the highest F1 scores of 97.77% and 98.21% achieved by LR and SVM, respectively.Oriola and Kotzé [42] introduced SVM, LR, RF, and gradient boosting algorithms for hate speech discovery based on South African Twitter data.The optimized SVM n-gram feature vector has obtained the highest TP rate of 0.894 [42].Herein, we propose an LSTMbased method for hate speech detection, which achieved 97% accuracy.Therefore, the developed LSTM method performed  superior to other contemporary ML approaches, including NB, SVM, LR, XGB, RF, k-NN, ANN, and BERT that obtained accuracies of 92%, 93%, 93%, 95%, 92%, 94%, 94%, and 93%, respectively.Initially, we prepared the dataset by extracting real-time tweets using Twitter API called REST API, which proved to be more diverse compared to datasets produced in related works.Our approach employs TF-IDF vectorization for the feature representations, which is superior to the count vectorizer method.The resulting visual frame of hateful tweets was generated using the word cloud method and reveals that the proposed method could effectively distinguish between hateful and nonhateful tweets.

VI. CONCLUSION AND FUTURE WORK
In this work, we propose an LSTM-based approach to detect hateful speech from a real-time tweet stream on Twitter using specific hashtags.The TFID mechanism applies vector representation that attaches weights to the meaningful terms associated with hateful or nonhateful tweets.The word cloud technique depicts the words responsible for hateful sentiments on a visual frame.The efficiency of our proposed approach to detect hate speech was compared to other reported methods, including NB, SVM, LR, XGB, RF, k-NN, ANN, and BERT.Results show that LSTM obtained the highest accuracy of 97%.In future studies, we aim to extend our approach for languages other than English, such as Bengali, Oria, Marathi, Tamil, and Telugu.Also, we plan to develop a hybrid model composed of advanced ML methods, evolutionary computation, and swarm intelligence to enhance the accuracy of hate speech detection for multilingual tweets.

Fig. 1 .
Fig. 1.Architecture of the hate speech detection method.

Algorithm 1
Calculation of TF-IDF Score 1 Input: an array of sentences.2 Output: TF-IDF score 3 Begi n 4 Import TF-IDF Vectorizer Module from sklearn 5 Create an array containing all tweets 6 Perform fit_transform on the array and store the output in a matrix 7 end C. Word Cloud Visualization for Prominent Hateful Words

Fig. 2 .
Fig. 2. Hate speech detection model using training and test data.

Fig. 3 .
Fig. 3. X -axis represents a number of characters a tweet, and Y -axis represents the number of tweets.

Finally, we
performed both unigram and bigram feature extraction on real-time tweets considering the length of the tweets, as shown in Figs. 6 and 7, respectively.

TABLE I PREVIOUS
WORKS ON HATE SPEECH DETECTION