Enhancing Hateful Meme Detection via Modality Enhancement and Multi-View Fusion

Memes, defined as a combination of visual and textual elements, have become a pervasive cultural phenomenon on the Internet. Some memes contain offensive content, which can have a significant impact on social media environments. The challenge of detecting hateful memes in a multimodal context is compounded by the compact nature of text and images, which differ in their semantic properties. In addressing this challenge, we propose a methodology grounded in the CLIP model’s dual-tower architecture, encompassing textual inversion and progressive learnable prompt strategies to enhance multimodal representations. Furthermore, visual representations are enhanced by activating attention pooling. For the purpose of modality fusion, we propose a multi-gate mixture expert network with an attention mechanism to efficiently refine and fuse modalities, dynamically adjusting weights for optimal classification. The efficacy of the proposed method is demonstrated by its superior performance in comparison to state-of-the-art techniques on four benchmark datasets for hateful meme detection.