Multimodal Affect Recognition Using Temporal Convolutional Neural Networks

The field of affective computing has gained significant attention from researchers in the last decade due to the wide variety of applications that can benefit from the technology. Nonetheless, recognizing spontaneous and subtle emotions remains a challenging problem for computers. In this work, we present a model for automatically recognizing human emotions using audio and video features. As opposed to most of the recent work that employs Recurrent Neural Network (RNN) models for emotion recognition from audio and video, we applied a Temporal Convolutional Network (TCN) to model the time-series information. To compare the RNN- and TCN-based networks, we configured the hyperparameters of the models using Gaussian processes and Bayesian analysis to achieve a fair evaluation. Our experimental results show that our TCN-based model outperforms all tested RNN-based models, yielding a concordance correlation coefficient of 0.7440 (vs. 0.7269) on valence and 0.7557 (vs. 0.7255) on arousal for the SEWA dataset.