Extending HoloGAN by Embedding Image Content into Latent Vectors for Novel View Synthesis

This study aims to further develop the task of novel view synthesis by generative adversarial networks (GAN). The goal of novel view synthesis is to, given one or more input images, synthesize images of the same target content but from different viewpoints. Previous research showed that the unsupervised learning model HoloGAN achieved high performance in generating images from different viewpoints. However, HoloGAN is less capable of specifying the target content to generate and is difficult to train due to high data requirements. Therefore, this study proposes two approaches to improve the current limitations of HoloGAN and make it suitable for the task of novel view synthesis. The first approach reuses the encoder network of HoloGAN to get the corresponding latent vectors of the image contents to specify the target content of the generated images. The second approach introduces an auto-encoder architecture to HoloGAN so that more viewpoints can be generated correctly. The experiment results indicate that the first approach is efficient in specifying a target content. Meanwhile, the second approach method helps HoloGAN to learn a richer range of viewpoints but is not compatible with the first approach. The combination of these two approaches and their application to service robotics are discussed in conclusion.