Be-or-Not Prompt Enhanced Hard Negatives Generating For Memes Category Detection

Memes are one of the most popular social media in online disinformation campaigns. Their creators often use a variety of rhetoric and psychological skills to achieve the purpose of misinformed audiences. These characteristics lead to the unsatisfactory performance of memes category detection tasks, such as predicting propaganda techniques, being harmful or not, and so on. To this end, we propose a novel memes category detection model via Be-or-Not Prompt Enhanced hard Negatives generating (BNPEN). Firstly, our BNPEN is reformulated into a contrastive learning-based image-text matching (ITM) task through category-padded prompt engineering. Secondly, we design the be-or-not prompt templates to keep the writing style of memes and create hard negative image-text pairs. Finally, our negatives generating can alleviate the negative-positive-coupling (NPC) effects in contrastive learning, thus improving the image-text matching quality. Conducted on two public datasets, experimental results show that our BNPEN is better than the off-the-shelf multi-modal learning models in terms of F1 and Accuracy measures.