Analysis of Eye Fixations During Emotion Recognition in Talking Faces

Research on emotion recognition from cues expressed in facial expression has a long-standing tradition. In this study, we investigate human’s visual attention and fixation patterns when identifying six basic emotions on expressive talking faces. Stimuli for the current experiments consisted of 92 video clips of facial expression during talking. The whole experiments were divided into two sessions. The video stimuli in the first session were presented in random order across different face identities, while in the second session the video from the same face identity were be played sequentially. The participants’ eye movements were recorded by the Tobii X3-120 screen-based eye-tracking system. We defined a set of area-of-interest (AOI) regions, including 4 AOIs of general face areas and 12 AOIs related to specific Action Units (AUs) involved in the coding of the six basic emotions. The gaze pattern analysis was done by looking at the fixation time on this predetermined set of AOIs. Based on the ANOVA analysis, we did not find significant differences in mean fixation time on any AOI for discriminating the six basic emotions, but a subset of significant AOIs was found when we sectioned the six basic emotions into positive, negative, and neutral. Next, we propose to develop a novel emotion perception classifier which can automatically classify an observer’s emotion perception based on her gaze patterns and fixation sequence when identifying the basic emotions on expressive talking faces. The fixation time on the 16 predetermined AOIs were used as features to train support vector machine (SVM) models. The proposed models achieved the overall classification accuracy of 84.1% on recognizing 3-way emotions of negative, positive and neutral, suggesting that the proposed eye gaze patterns - the fixation time on 16 predetermined AOIs, are very promising for automatic classification of the perceived emotions. Finally, we divided the data into different gender and race groups, and discussed the diversity in gaze patterns across genders and different race groups.