A temporally coherent neural algorithm for artistic style transfer

Within the fields of visual effects and animation, humans have historically spent painstaking hours mastering the skill of drawing frame-by-frame animations. One such animation technique that has been widely used is called “rotoscoping” and has allowed uniquely stylized animations to capture the motion of real life action sequences. Automating this arduous process would free animators from performing frame by frame stylization to concentrate on artistic contributions. We introduce a new artificial system based on an existing neural style transfer method which creates artistically stylized animations that simultaneously reproduce both the motion of the original videos that they are derived from and the unique style of a given artistic work. This system utilizes a convolutional neural network framework to extract a hierarchy of image features used for generating images that appear visually similar to a given artistic style while at the same time faithfully preserving temporal content. The use of optical flow allows the combination of style and content to be integrated directly with the apparent motion over frames of a video to produce smooth and visually appealing transitions. This implementation demonstrates how biologically-inspired systems such as convolutional neural networks are rapidly approaching human-level behavior in tasks that were once thought impossible. Further, this research provides unique insights into the way that humans who produce artistically stylized animations perceive temporal information.