Multi-task deep neural network with shared hidden layers: Breaking down the wall between emotion representations

Emotion representations are psychological constructs for modelling, analysing, and recognising emotion, being one essential element of affect. Due to its complexity, the boundaries between different emotion concepts are often fuzzy, which is also reflected in the diversification of emotion databases, and their inconsistent target labels. When facing data scarcity as an ever present issue for acoustic emotion recognition, the straightforward method to jointly use the existing data resources is to map various emotion labels onto one common dimensional space; this, however, comes with considerable information loss. To solve the dilemma of data aggregation whilst efficiently exploiting the emotion labels in terms of their original meaning and interrelations, we advocate the usage of multi-task deep neural networks with shared hidden layers (MT-SHL-DNN), in which the feature transformations are shared across different emotion representations, while the output layers are separately associated with each emotion database. On nine frequently used emotional speech corpora and two different acoustic feature sets, we demonstrate that the MT-SHL-DNN method outperforms the single-task DNNs trained with only one emotion representation.