Hierarchical Context Interaction and Reasoning with Transformer for Emotion Recognition

Emotion recognition is an important task in computer vision. However, current approaches using hard associations (e.g., element-wise addition or concatenation) suffer from information pollution. To overcome these challenges and utilize information at different scales, we present a novel Transformer-based emotion recognition (TransEmo) framework utilizing only the RGB modality as input. TransEmo is a hierarchical feature fusion framework with soft-association to effectively realize contextual interaction and emotion reasoning. It comprises a top-down path and a bottom-up path with cross-attention layers. The top-down path enriches the features of the current hierarchy gradually by using information from a larger receptive field and interacts effectively with contexts to suppress redundant information and noise. The bottom-up path introduces a set of learnable prototypes to adaptively capture clues for various emotions by reasoning with hierarchical features. Our proposed approach achieves state-of-the-art performance on two benchmarks and provides an analysis to discuss the advantages of our modules.