Leveraging valence and activation information via multi-task learning for categorical emotion recognition

Deep learning technologies have been successfully applied to acoustic emotion recognition lately. In this work, we propose to apply multi-task learning for acoustic emotion recognition based on the Deep Belief Network (DBN) framework. We treat the categorical emotion recognition task as the major task. For the secondary task, we leverage two continuous labels, valence and activation. Two strategies are employed to achieve multi-task learning. First, we map the continuous labels into three categorical labels: low; medium; high, and use classification for the secondary task. Second, we project the continuous labels into [-1; 1] range, and use regression for the secondary task. The combination of the loss functions from the major and secondary tasks is used in the objective function in multi-task learning. After iterative optimization, the values from the last hidden layer are used as features in the backend SVM classifier for emotion classification. Our experimental results show significant improvement over the baseline results using DBN, suggesting the benefit of utilizing additional information in a multi-task learning setup.