A Multimodal Deep Learning Approach to Emotion Detection and Identification

Automated emotion recognition and identification and its subsequent challenges have a long history. More recently, intense scientific research on computer based evaluation of human emotions has arrived at a crossroad. Reputable scientists in the cognitive science domain consider that the system built on Ekman's seven basic emotions is vitiated by generalizations obtained on a reduced number of test cases. In contrast, computer scientists consider that the progress made so far in the theory and application of Neural Networks allows computers to increase the accuracy of emotion detection and identification. A Multimodal Convolutional Neural Network (MMCNN) for emotion detection and identification in near real-time, will be introduced in this paper. The MMCNN detects, identifies and tracks users' emotions, by reasoning on facial micro-expressions, on body motions and on speech. A CNN classifies the emotion into one of the 7 universal classes accepted so far. The deciding classifier then takes the scores generated from both the micro-expression detector and speech synthesizer to predict the emotion. The emotion class is validated using the Berkeley Expressivity Questionnaire. Results on testing the accuracy of the algorithm are given at the end of this paper.