MEAS: Multimodal Emotion Analysis System for Short Videos on Social Media Platforms

Short videos have surged in popularity on social media. The emotions expressed by short videos can trigger or even magnify the public sentiment. Hence, accurate computation of these emotions is vital for social affective computing. However, the multimodal emotion analysis of short videos on social platforms faces some challenges: the accuracy of the model is tested by the disunity of video resolution; the collection of large-scale social media data, the manual transcription and segmentation of audio content, and the precise labeling process require a lot of manpower. In this article, we have proposed an affective computing system MEAS for social short videos, which combines multiscale resolution adaptability and advanced RoBERTa model to optimize the preprocessing of high definition large size short videos and improve the contribution of text modality in emotion analysis. In addition, the system also adopts automatic audio segmentation and transcription technology to realize the efficient capture of speech forms in social short videos. Experimental results show that compared with the leading open source algorithm V2EM on the IEMOCAP dataset, the proposed method achieves a significant increase in weighted accuracy and F1 score of 4.17% and 7.29%, respectively. We constructed a novel dataset named “Bili-news” based on social platform news short videos, validating the effectiveness of the MEAS system. Through experimental verification, we also find a significant positive correlation between the emotions expressed in short videos and the social sentiments of the audience.