Classification of Hateful Memes by Multimodal Analysis using CLIP

Due to the widespread usage of social media platforms, memes and other hateful contents have become a major concern as they propagate harmful stereotypes, and often inciting violence and social tensions in societies. To counter it, this research paper proposes a novel approach for the detection of such hateful memes using multimodal techniques by using the CLIP (Contrastive Language-Image Pre-training) model. As the image and textual parts of the memes are seldom semantically aligned, thus we need to integrate both these information to capture the correct context of the memes to detect the hateful memes. Through extensive testing on a large dataset of memes provided by Meta we demonstrate the accuracy of this approach. Through this paper we contribute to the development and refinement of more robust and ethical solutions to tackle this growing problem of hate speech to promote a healthy and safe discourse on social media.