Real Time Detection of Learner Fatigue in E-Learning Environments through the Combination of Eye and Mouth Features

one of the features of intelligent e-learning environments is the presence of a personalized mechanism to assist the effective learning for each learner. Exploiting personalized tutoring models, learners can experience successful and satisfying learning outcomes due to their learning objectives. In this way, the learner (student) feels that there is always an aware and insightful tutor (teacher) alongside, providing private guidance on the learning path. While personalizing the learning environment is highly desirable, the continuous monitoring of the virtual teacher may lead to fatigue (and even disillusionment) for the learner, potentially reducing the efficiency of the e-learning environment. Considering the above points, in this article an intelligent system has been designed to identify fatigue in learners by examining changes in their facial expressions. To achieve this, a combination of eye and mouth features of the learner is utilized, and with the help of a deep neural network, the level of learner fatigue is detected. The proposed system employs the Media‌Pipe algorithm for face and eye-mouth feature identification, and for training the neural network model, the Efficient‌Net neural network has been utilized. In this system, databases NITYMED and Drowsiness‌Detection have been utilized for training the network. Additionally, 80% of dataset has been used for training and the remaining 20% for testing. It is worth mentioning that given the balanced distribution of samples for each class, accuracy is employed as the evaluation metric, considering the objective of minimizing false positives. The experimental results show the accuracy and precision of 99.73% and 99.72% for the eye model, respectively, and the same measures for the mouth model are 99.85% and 99.95%, respectively. The standard deviation value for the eye model is 0.4997, and for the mouth model, it is 0.4998, indicating that most of the data points are close to the mean. Additionally, the mean value interval for the eye and the mouth model are [0.44581166, 0.55418825] and [0.5688507, 0.43114927], respectively and suggesting that the models have achieved good uniformity in predicting classes. The designed system has been implemented in a e-learning environment, and the experimental results demonstrate accurate fatigue detection in the subjects.