Sentiment Classification of Multi-modal Memes Using Deep Learning and Transformer Techniques

The widespread use of social media has elevated memes to a powerful mode of online communication. However, their misuse can fuel negativity and harm individuals’ mental health. Recognizing the sentiment of memes has become a challenge due to their multi-modal nature. This study proposes a method for sentiment analysis of Bengali memes, leveraging the concepts of multimodality. We investigated various deep learning techniques (CNN and BiLSTM) for textual features (ResNet50, VGG16, and VGG19) for visual features, as well as transformer techniques (m-BERT, XLM-R, and Bangla-BERT for textual features, and Vision Transformer and ConvNeXT for visual features). By combining image and text, our ConvNeXT+m-BERT model Proved highly effective, attaining an impressive f1-score of 0.712. Our proposed model (ConvNeXT+m-BERT) showed the highest performance on the ‘MEMOSEN’ [1] dataset compared to previous models. In a world where memes wield significant influence, our approach offers a crucial step toward understanding and managing their impact on digital conversations and mental well-being.