Generate Event Schemas from Multimodal Corpus

Event schemas are conceptualized and structured knowledge organizations that generalize schemas in a class of events. The core issue in event schema research is how to generate high-quality and comprehensive event schemas automatically. Existing methods focus on generating event schema from the textual corpus and fail to exploit the complementary advantages of multimodal information effectively. Therefore, this paper proposes a method to generate event schemas from a multimodal corpus of images and text. It utilizes the ability of a multimodal large language model to generate text from images to supplement additional event information and then uses the in-context learning of large language models to generate conceptualized and structured event schemas. Experimental results show that the proposed method can generate more comprehensive event schemas and improve the knowledge of missing event schemas in the visual modality.