Multimodal Emotion Recognition in Video, Audio, and Text using Deep and Transfer Learning

Emotions play an important role in an efficient communication process. Many researchers have studied emotion recognition, however, the majority of their work has concentrated on one modality like text, or numerous modalities like text and speech, but just for one language. This imposes a restriction on the ability to recognize speech, text, or video. Considering multiple modalities at once is the answer to this. There is little study that takes into account all three modalities simultaneously. Therefore, in this paper, we propose a multimodal approach that detects the emotion expressed by a user in a video input where we will not only analyze the facial expression but also take the speech and the text into account while predicting the emotion and expand it to cross-language scenarios. The video input taken detects facial structure and the respective emotion by using some well-known deep learning models. For speech emotion prediction we use three datasets out of which one of them is a different language to implement the cross-language and cross-corpus scenario. The text extracted from audio is given as input to emotion recognition models within the Machine Learning and Deep/Transfer Learning domains. Our results suggest that the multimodal system has a lot of credibility due to the consideration of all 3 modalities and give us a much more accurate context or breakdown of the emotions interpreted from the user input.