QoE Assessment for Prompt-based Videos: Acceptability and Quality Factors

The development of video technology and consumer devices has resulted in a boom of user-generated clips on various social networking services. The recent OpenAI Sora model can revolutionize the production of consumer videos, simply using prompts. However, videos generated using prompts usually contain new artifacts induced by generative AI. In addition, while various researchers have conducted extensive quality assessments on AI-generated videos, studies specifically focused on the Quality of Experience (QoE) of such videos remain limited. Given the increasing role of generative AI in media production, it is crucial to evaluate how users perceive and interact with these videos. In this paper, we present the first study on the acceptability assessment (also known as Customer Satisfaction Score) of prompt-based videos, which are generated by OpenAI Sora. In the assessment, subjects were asked to rate the video's alignment, perceptual quality, aesthetic quality, and whether the video is acceptable or not. The results show that, though the videos are visually impressive, not many videos are actually acceptable to users. Also, the relationship between acceptability and the other quality aspects is complex and non-linear. Especially, it is found that perceptual quality contributes the most to acceptability, while alignment is the least significant contributor. In addition, we identify over ten quality factors that cause low acceptability of prompt-based videos. Lastly, results from objective evaluation show that the performances of current quality metrics are insufficient to automatically monitor the quality and alignment of prompt-based videos.