ProMNER: Prompt-Guided Unimodal and Multimodal Named Entity Recognition in a Generative Way

Multimodal named entity recognition (MNER) on social media is a challenging task that uses corresponding images to aid in recognizing entities in short and noisy texts. Most existing methods suffer from two major drawbacks: weak image-text semantic correlation and the hybridity of entity types in intermodality alignment. To this end, a general framework ProMNER is proposed for unimodal and multimodal NER tasks, which includes text-driven visual-modality enhancement (TVE) and prompt-guided type-related refinement and interaction (PTRI). Specifically, to alleviate the bias caused by irrelevant visual content, TVE employs cross-modal translation to generate visual representations that replace the original ones. Furthermore, the generated can serve directly as an additional input modality for NER, enabling the framework to be generalized to unimodal and multimodal NER tasks. Subsequently, to tackle the hybridity problem of intermodality entity alignment, type-filled question prompts are designed as guidance information to explicitly refine type-related visual representations. PTRI obtains type-specific multimodal representations by capturing semantic intermodality interactions for each type, and learns a comprehensive multimodal representation through dynamic weighting. Experimental results on five benchmark datasets demonstrate the superiority and generality of ProMNER in unimodal and multimodal NER tasks.