Constrained Manifold Learning for Videos

Automatic image manipulation can be used to make subtle changes at the pixel level resulting in morphism from one domain to another. This is desirable in tasks such as creating mock expressions for an individual or dynamic scene generation in autonomous driving. This type of morphism can be achieved using an adversarial model where the generator and the discriminator compete to produce fake images of the target domain. Due to high variance among the images, it is difficult to learn an optimal loss function. Previously, manifold matching of clusters in the source domain with labeled samples and the target domain that is generated was used to overcome this limitation. To generate videos it is common to use three-dimensional convolution however, such a model has very high complexity. Instead, in this paper we use manifold constrained model selection to do a constrained clustering of the combined manifold with fixed start and end images for the morphism. We show that each step in the principal path connecting the centroids is analogous to a single time delay in the video sequence. Hence, we can construct a cascade of models using samples from a pair of connected centroids such that one model is used to initialize the next. We apply the model to smile generation from neutral face expression and for predicting the next few frames while driving on real roads. We are able to outperform the baselines in the quality of images generated and the computational cost for training the model.