A Neural Network Approach to Score Fusion for Emotion Recognition

Automatic facial emotion recognition has been one of the interesting research topics in the recent decades. There have been recent advances in convolutional neural networks (CNNs) which have become the state-of-the-art approaches in pattern recognition. This paper presents an effective facial emotion recognition system that uses convolutional neural networks (GoogleNet-CNN) for eyeglass detection and feature extraction followed by a novel score fusion model. The proposed system has three key components for improving recognition performance: 1) a highly accurate glasses detector to differentiate between images in which the human subject is wearing glasses and images in which the human subject is not wearing glasses, 2) convolutional neural networks to extract nine different sets of emotional features from faces with and without glasses, which are then classified by support vector machines (SVMs), 3) a multiple classifier system (MCS) to accomplish decision fusion by using a neural network. The USTC-NVIE (NVIE) database is used to evaluate the performance of the proposed system. Experimental results show that the eyeglass detector obtained a best overall accuracy of 99.7% and the proposed facial emotion recognition system can achieve 7-15% higher classification rates when using the eyeglass detector. By applying the neural network approach in the multiple classifier system for score fusion, the classification rates of the system increase by about 10%.