True Feature Distillation for Compression of GANs

The ResNet architecture based Pix2Pix model has a high number of dependent operations owing to the numerous ResNet Blocks. Thus, during the training and inference of the model, a high latency is observed even with powerful GPUs. More layers reduce the number of independent operations, thus underutilizing the high parallelism available in powerful GPUs. The proposed method of True Feature Distillation (TFD) implements GAN compression based on reducing layers for compression along with reducing the filters for increasing the parallel operations in the model and in turn increasing the usage of parallelism of powerful GPUs. Through TFD, a reduced latency is observed with negligible change in performance of the model even with a reduced number of layers in the proposed model. A ResNet based Pix2Pix model is used as an example for training through the proposed method via edges2shoes dataset. Speedup with respect to complete Pix2Pix achieved is 1.58x on the NVIDIA Tesla V100.