Automatic Generation of Multimedia Teaching Materials Based on Generative AI: Taking Tang Poetry as an Example

Generative artificial intelligence (AI) is widely recognized as one of the most influential technologies for the future, having sparked a paradigm shift in scientific research. The field of education has also been greatly impacted by this transformative technology, with researchers exploring the applications of generative AI, particularly ChatGPT, in education. However, existing research primarily focuses on generating text from text, and there remains a relative scarcity of studies on leveraging multimodal generation capabilities to address key challenges in multimodal data supported instruction. In this article, we present a technical framework for generating Tang poetry situational videos, emphasizing the utilization of generative AI to address the need for multimedia teaching resources. Our framework comprises three main modules: textual situational comprehension, image creation, and video generation. Moreover, we have developed a situational video generation system that incorporates various technologies, including text-to-text generation models, text-to-image generation models, image interpolation, text-to-speech synthesis, and video synthesis. To ascertain the efficacy of the modules within the Tang poetry situational video generation system, we undertook a comparative analysis utilizing the prevalent text-to-image and text-to-video generation models. The empirical findings indicate that our approach is capable of generating images that exhibit greater semantic similarity with the poems, thereby enabling a better comprehension of the poem's connotations and its key components. Concurrently, the Tang poetry videos generated can significantly contribute to the reduction of cognitive load and the enhancement of understanding during the learning process. Our research showcases the potential of generative AI in the education field, specifically in the domain of multimodal teaching resources.