Cinematographic-Aware Coherent Shot Assembly for How-To Vlog Generation

The how-to vlog has gained popularity as a medium for learning practical skills, such as cooking and handcrafting. The sequencing process of these videos should meticulously integrate storytelling and cinematographic elements to ensure audience comprehension and engagement, posing a significant challenge for novice creators. Pioneer efforts have adhered to predefined editing rules or assembled clips based on textual logic, limiting their applicability across diverse educational scenarios and causing visual discontinuities. In contrast, we aim to extract rich cinematographic patterns from well-edited professional how-to vlogs to enhance shot assembly. To this end, we identify two crucial aspects influencing educational value: narrative continuity and scale transition. We model the shot assembly process as a task of selecting the next shot and devise a cinematographic-aware contrastive model to learn representations that distinguish between the good next shots against the bad ones. This method incorporates a two-stream cinematographic-aware encoding module for explicit factor encoding and a situation-adaptive attention-based integration module to accommodate varying assembly scenarios. Quantitative results from our novel professional user-generated vlogs dataset (proVlog-HowTo) clearly demonstrate the proposed method’s effectiveness. User study results further indicate its superiority in generating videos with narrative continuity and smooth transitions.