Meme Analysis Using LLM-Based Contextual Information and U-Net Encapsulated Transformer

A meme is social media content with which the creator tries to convey a certain idea in public via the internet. Each meme consists of typically an image and supporting text. Its message can be humorous and inspiring, but hilarious and offensive often targeting a specific audience. To address the potential harm such memes can cause, Artificial Intelligence researchers have proposed solutions to classify a meme automatically according to the sentiment, emotion, and intensity felt by the users. Recent models for meme analysis often adopt the Transformer architecture, which is known to perform well but computationally expensive. The present study aims to introduce a novel method by providing (1) deep contextual information and (2) reducing resource utilization while keeping its efficiency. For the former, GPT-4 has been utilized to provide meaningful insights regarding the context behind the meme. For the latter, we extract Keyphrases and forward them to a U-net Encapsulated Transformer, called UET, to process the information. Extensive evaluation with ablation study using three standard meme datasets, i.e. Memotions, suggests that it outperforms state-of-the-art models on sentiment analysis, while it shows comparable performance on the emotion and intensity task. As the proposed model is more lightweight than a standard one and yet shows high performance, it provides new insights into meme analysis and could be useful for other Natural Language Processing tasks.