Sarcasm Detection on Image Data Using Multimodal Approach

Sarcasm detection presents a complex challenge in both natural language processing and computer vision, as it requires understanding subtle contextual cues, tone, and the relationship between literal and intended meanings. Traditional approaches often rely on either text or visual data alone, limiting their ability to effectively detect sarcasm due to its nuanced nature. Sarcasm frequently arises from the interaction between textual and visual elements, where both provide essential context that cannot be fully understood in isolation. To address this, we propose a novel multimodal approach for sarcasm detection, combining textual and visual information to capture these interactions more effectively. Our method uses a carefully curated dataset of images and captions or tweets, extracting visual features through ResNet and textual features with BERT. This integrated feature set allows the model to better understand the interplay between text and visuals. We train the model using Artificial Neural Networks (ANN) and Support Vector Machines (SVM), evaluating its performance with metrics such as accuracy, precision, recall, and F1-score. Our experimental results demonstrate that the proposed multimodal framework significantly outperforms traditional unimodal approaches, highlighting the effectiveness of combining both textual and visual cues for more accurate sarcasm detection, particularly in applications such as social media analysis and content moderation.