Nested Variance Estimating VAE/GAN for Face Generation

We study the task of conditional image generation and provide a general framework for combining autoencoders (AEs) with generative adversarial networks (GANs). Our framework provides a principled way to avoid well-known problems such as mode collapse and training instability. We use two AEs, a big parent-AE and a small child-AE, to play different roles. Our parent-AE is trained to minimize only one single objective: the reconstruction loss, which makes its training process stable and efficient. It is then fixed and used for several purposes, including initializing the generator and providing powerful features for a simple discriminator of GAN. It also plays the role of reducing the initial harder task of image generation to a simpler one: sampling from its latent distribution, which is given to child-AE to accomplish. Child-AE can be trained very efficiently due to its small size, and we only need to modify it if necessary for different applications, which makes our framework very flexible. Our experiments show that our model is capable of generating high quality novel images with controllable attributes.