Video Stylization: Painterly Rendering and Optimization With Content Extraction

We present an interactive video stylization system for transforming an input video into a painterly animation. The system consists of two phases: a content extraction phase to obtain semantic objects, i.e., recognized content, in a video and establish dense feature correspondences, and a painterly rendering phase to select, place, and propagate brush strokes for stylized animations based on the semantic content and object motions derived from the first phase. Compared with the previous work, the proposed method has the following three advantages. First, we propose a two-pass rendering strategy and brush strokes with mixed colors in order to render expressive visual effects. Second, the brush strokes are warped according to global object deformations, so that the strokes appear to be naturally attached to the object surfaces. Third, we propose a deferred rendering and backward completion method to draw brush strokes on emerging regions and simulate a damped system to reduce stroke scintillation effect. Moreover, we discuss the graphics processing unit-based implementation of our system, which is demonstrated to greatly improve the efficiency of producing stylized videos. In experiments, we verify this system by applying it to a number of video clips to produce expressive oil-painting animations and compare it with the state-of-the-art approaches.