Group-Level Emotion Recognition Using Hierarchical Dual-Branch Cross Transformer with Semi-Supervised Learning

Group-level emotion recognition (GER) has received attention from researchers to identify an overall emotion in a multi-person scene. To address attention issue on group dynamic in GER, a dual-branch cross-attention transformer was recently developed. However, the existing method failed to effectively distinguish confused emotion categories. On the other hand, the data scarcity of GER dataset limits the performance improvement of GER. In this paper, we propose a novel method,namely hierarchical dual-branch cross transformer with semi-supervised learning for GER. Firstly, we introduce semi-supervised learning to fully exploit the unlabelled data for dual-branch cross transformer architecture. Furthermore, hierarchical classification stage is presented in the architecture, where two scene information with various patch size was used in the first stage, while scene and the most important face information in second stage. Intensive experiments are conducted on group-level emotion database. The experimental results demonstrate that the proposed approach achieves promising performance for GER compared with the recent state-of-the-art methods.