Accurate Identification of Human Emotional States from Images Using Deep Learning

Facial expression recognition is a crucial aspect of human communication, especially for building social relationships. However, machine-based recognition remains a challenging task where implemented SOTA methods achieved testing accuracies from 50% to 60% on FER-2013. Our research proposes an automatic emotion identification system that utilizes emotional state heatmaps (ES-MAPs) and neural network classification algorithms. Using MediaPipe Face Mesh, our system extracts facial landmark coordinates and calculates the distance between all landmarks. A neutral baseline is subtracted from the landmark distances and saved as a heatmap to train a designed CNN model. The use of ES-MAPs minimizes noise introduced by significant variation in image lighting conditions and variation in head rotation. Our proposed system, ESH-Net, achieved 75% in test accuracy, 15% greater than that of other state-of-the-art methods on the HDFE dataset. In addition, ES-MAPs produced better clustering than the original facial images, indicating significant improvement in the separability and consistency of representation of emotional states. This study demonstrates the potential for emotional state heatmaps and deep learning models to significantly improve the accuracy and efficiency of emotion identification, which can greatly assist in assessing patient’s emotional state in medical diagnosis and practice.