Emotion Recognition of Images Based on Label Enhancement for Consumer Electronics Applications

Consumer electronics have gained widespread popularity and generated massive data, such as text, speech, and images. Different from other modalities, images have richer semantics, among which emotion is one of the most interesting characteristics, which can empower various consumer electronics applications, such as depression diagnosis, health monitoring, and recommendation. Generally, images, especially those generated by consumer electronics (i.e., smartphone) in real-world scenes, have blended emotions. Literature has proposed to capture the blended emotions with emotion distribution that covers all emotions with different relevance. However, images are generally annotated with logical labels without emotion distribution. In this paper, we apply the label enhancement (LE) technique to emotion recognition of images. We design a novel LE algorithm by exploiting label manifold (LELM) to enhance the logical labels into emotion distributions. After obtaining the emotion distributions, we train a deep neural network with KL-divergence and margin loss to learn such emotion distributions. Finally, we conduct experiments on four large-scale image datasets. The experimental results justify that our method achieves the best results for enhancing emotion distributions and highest accuracy for emotion prediction.