Multimodal Event Classification for Social Media Based on Text-Image-Caption Assisted Alignment

The vast amount and diverse forms of information (such as text, images, etc.) provide people with rich data. How to effectively obtain and utilize multimodal data has gradually become a research hotspot in the field of artificial intelligence. Taking text-image pairs in social media, multiple different methods were adopted to achieve text and image feature fusion, achieving more robust event classification. A multimodal event classification model (A-MECM) based on attention mechanism had been proposed. A-MECM constructed the multimodal attention module to achieve mutual attention for multimodal event classification. Furthermore, a multimodal event classification model (ACS-MECM) utilizing caption-semantics of images to assisted alignment was proposed. Firstly, ACS-MECM utilized a Vision-Language Pre-training (VLP) Model - Oscar - to generate caption for images in the dataset, in order to obtain the semantics of the images; secondly, a Pre-training Language Model (BERT) and a Pre-training Vision Model (Vision Transformer, ViT) were used to extract features from text and images, respectively; finally, a cross-attention encoder was constructed to utilize image caption to achieve feature fusion between text and images. Classifying text-image pairs of specific domain events (such as disasters) in social media can achieve preliminary screening of data information, achieve coarse-grained classification, and provide assistance to relevant departments and government agencies.