Decoupled Learning for Conditional Adversarial Networks

Incorporating encoding-decoding nets with adversarial nets has been widely adopted in image generation tasks. We observe that the state-of-the-art achievements were obtained by carefully balancing the reconstruction loss and adversarial loss, and such balance shifts with different network structures, datasets, and training strategies. Empirical studies have demonstrated that an inappropriate weight between the two losses may cause instability, and it is tricky to search for the optimal setting, especially when lacking prior knowledge on the data and network. This paper gives the first attempt to relax the need of manual balancing by proposing the concept of decoupled learning, where a novel network structure is designed that explicitly disentangles the backpropagation paths of the two losses. In existing works, the encoding-decoding nets and GANs are integrated by sharing weights on the generator/decoder, thus the two losses are backpropagated to the generator/decoder simultaneously, where a weighting factor is needed to balance the interaction between the two losses. The decoupled learning avoids the interaction and thus removes the requirement of the weighting factor, essentially improving the generalization capacity of the designed model to different applications. The decoupled learning framework could be easily adapted to most existing encoding-decoding-based generative networks and achieve competitive performance without the need of weight adjustment. Experimental results demonstrate the effectiveness, robustness, and generality of the proposed method. The other contribution of the paper is the design of a new evaluation metric to measure the image quality of generative models. We propose the so-called normalized relative discriminative score (NRDS), which introduces the idea of relative comparison, rather than providing absolute estimates like existing metrics.