An Efficient Approach for Audio-Visual Emotion Recognition With Missing Labels And Missing Modalities

Audio-visual emotion recognition is important for human-machine interaction systems by combining the information of audio and visual modalities. Although great progress has been made by previous works using multimodal learning compared with unimodal learning, they still cannot effectively deal with two key challenges. Firstly, it is difficult or expensive to acquire labeled emotional data, which results in a large amount of data with missing labels. Secondly, emotional data often has missing modalities. To address these problems, we propose a unified deep learning framework to efficiently handle missing labels and missing modalities for audio-visual emotion recognition through correlation analysis. Specifically, we consider four types of emotional data during the training stage: complete, label missing, visual missing, and audio missing. We propose a correlation loss based on Hirschfeld-Gebelein-Ŕenyi (HGR) maximal correlation to effectively capture the common information in different types of training data for emotion prediction. Experiments on the eNTERFACE’05 and RAVDESS datasets show that our deep learning approach has high effectiveness for audio-visual emotion recognition.