Happy Emotion Recognition in Videos Via Apex Spotting and Temporal Models

With rapid increasing research in emotion detection, facial expression recognition becomes more popular as a vital emotion measurement instrument. Happy is one of the most common facial expressions that indicates a positive human emotional status. Although many facial expression recognition methods have been well established for recognizing multiple emotions, studies on happy detection are very limited, especially for processing the videos. In this paper, we propose a new single emotion recognition method to recognize happy emotion from key frames of facial expression videos. In our method, we extract key frames that demonstrate the highest intensity level of expressions among all frames in two steps. In the first step, an Inception ResNet framework is utilized to segment an facial expression process in a video into three parts: onset, apex and offset. In the second phase, we choose first three frames in the apex segment as key frames. A fine-tuned convolutional neural network (CNN) then classifies the key frames into happy and non-happy classes. Our experimental results demonstrate that the proposed approach achieves higher accuracy than four counterpart methods in recognizing happy emotions, with the accuracy of 98.4% and 94.2% on two benchmark facial expression datasets, i.e., the extended Cohn-Kanade (CK+) and MMI.