Unsupervised Sketch-to-Image Translation Network for Single Category Image Synthesis

Sketch-based image synthesis is a challenging cross-modal translation problem in the field of computer vision. Existing supervised learning methods either rely on edge information or resort to category labeling. However, the amount of sketch-photo pair training data is limited. Accordingly, in this paper, we propose an unsupervised two-stage synthesis model to accomplish fine-grained sketch image translation at the pixel level. More precisely, we boldly use the grayscale image as an intermediate result of the synthesis task between the geometric transformation and the color filling process. In addition, we propose a new perceptual discriminator on the unpaired datasets to better learn the geometry of sketches, while incorporating an attention mechanism that preserves the main information of sketches to handle sketch-specific abstractions and stylistic variations. The final experimental results indicate that our approach achieves more promising results, obtaining more realistic and diverse generated images with significantly better FID scores.