A Novel Automatic Content Generation and Optimization Framework

With the rapid growth of IoT multimedia devices, more and more content is being delivered in multimedia forms which generally requires more effort and resources from users to create and could be challenging to streamline. In this article, we propose Text2Animation (T2A), a framework that helps to generate complex multimedia content, and animation, from the simple textual script input. By leveraging recent advances in computational cinematography and video understanding, the purposed workflow further reduces associated knowledge requirements dramatically for cinematography. By jointly incorporating the fidelity and aesthetic models, T2A jointly considers the comprehensiveness of the visual presentation of the input script and the compliance of generated video with given cinematography specifications. The virtual camera placement in a 3-D environment is mapped into an optimization problem that can be resolved by using dynamic programming to achieve the lowest computational complexity. Experimental results show that T2A can reduce the manual animation production process by around 74%, and the new optimization framework can improve the perceptual quality of the output video by up to 35%. More video footage can be found at https://www.youtube.com/watch?v=MMTJbmWL3gs.

I. Introduction

The exponential growth in the number of IoT multimedia devices with screens, cameras, and microphones brings significant opportunities to multimedia content distribution. In the past few years, tens of millions of video and game consumers have been transformed into media creators with the support of social Apps, such as Tiktok, Wechat, Roblox, Twitch, YouTube, and so on. Various IoT devices have been utilized during the content creation and sharing process. The IoT device makers are currently aggressively working with the service providers to add hardware and software supports into their systems to enable more advanced AI technology, such as computer vision, face recognition, and natural language understanding (NLP) for the demand of media generation.