Stationary dynamic texture synthesis using convolutional neural networks

In this paper, we utilize deep networks to study the problem of modeling and synthesizing stationary dynamic textures that exhibit both spatial and temporal regularity. Our work is a generalization of the generative static texture model based on convolutional neural networks (CNNs). We extend the static texture synthesis method using CNNs to the dynamical setting by adding the temporal dimension, where the multiple layers of spatial-temporal filters come from off-the-shelf deep 3-dimensional convolutional networks (3D ConvNets) trained in a discriminative fashion. Dynamic textures generation is formulated as an iterative optimization procedure which imposes the correlation statistics derived from 3D ConvNets of the input exemplar on the synthesized one. Synthesis experiments on various video texture exemplars show that such generative method can synthesize realistic dynamic textures of good visual quality.