Performance Comparison Between Sentiment Classification Models for Use in Chatbots in Virtual Learning Environments

This summary presents a complete article, and its category is research. With the growing number of conversational virtual assistants used in virtual learning environments (VLE), the humanization of these chatbots brings several benefits to their users, such as greater satisfaction with the service and assistance in the teaching-learning process, as shown by several studies cited in the work. For the humanization of these chatbots, it is necessary to classify the texts typed by the users, and for that, the best model must be used to identify the emotion in the message. This article presents tests developed to investigate, using Machine Learning, which classification model obtains the best performance for the proposed problem of classification of feelings. The experiments compared the accuracies between a recurrent neural network (RNN) and the Naive Bayes algorithm. The RNN was developed in Python using the Keras library. The developed model has approximately 300 neurons in the first layer and 128 in the second, totaling 640,000 and 130,000 parameters, respectively. In the third layer, three neurons were applied, leaving a total of 771,971 parameters to be estimated. Multinomial Naive Bayes was used, as it allows a non-binary output; it was implemented using the Scikit-Learn library. The alpha used was 0.001 because, after some adjustments to improve its result, it was the one that presented the best accuracy. Three datasets taken from Twitter were used to perform the tests. It followed (Wilson et al., 2019) as a basis for an acceptable accuracy result, where the minimum value is 85%. The recurrent neural network model presented an accuracy of 86.83% and the Naive Bayes algorithm 76.57%. After performing each experiment, the F1 Score metric was used through the Scikit-Learn library to confirm the results. With the analysis of the three experiments, it was observed that the neural network met the need for classifying feelings satisfactorily. At the same time, the Naive Bayes model did not reach the expected accuracy. The contribution of this article is to implement a model capable of recognizing the emotions present in the text when interacting with the chatbot to offer better service to users.