Fusion-based Model for Social Gathering Emotion Classification

In social events, emotion detection is critical for improving security by monitoring crowd behavior and recognizing potential hazards. This paper presents a novel approach for detecting and classifying group emotions in such scenarios, based on a newly developed EMO dataset of 442 images divided into four emotional categories: happy, angry, sad, and surprised. To increase classification performance, we created a fusion-based model that combines the advantages of a custombuilt Convolutional Neural Network (CNN) and a pre-trained Vision Transformer (ViT). This fusion technique combines the CNN’s ability to capture precise features with the ViT’s ability to understand global context, making the model more robust for emotion detection. Moreover, we fine-tune the model to attain the ideal configuration and compared to the VGG-19 model. However, Our model beat the benchmarks, with an accuracy of $\mathbf{7 1 . 5 9 \%}$ on validation data and $\mathbf{7 6 . 4 0 \%}$ on test data, as well as improved precision, recall, and F1-score measures. These findings highlight the fusion model’s effectiveness in real-time emotion recognition as well as its potential uses in improving social event security measures through accurate, timely emotional analysis of group dynamics.