Shrinking Neural Style Transfer Model with Knowledge Distillation

Neural style transfer is a technique to transfer a style of an artistic image to another photorealistic image. The early development of this technique is exploiting the intermediate output of a deep learning model and utilizing it to optimize a targeted image. However, this technique needs some time to finish the process. An approach called knowledge distillation is widely used to optimize deep learning models so the speed of the model can be increased. Commonly, this technique is applied in a supervised task by providing a soft label as a target label for the student model. In this paper, a method is proposed to enable knowledge distillation happened in style transfer tasks. It is found this method is effective enough to make the style transfer process faster and give good results. In this paper also some discussion has been elaborated regarding the condition that is adequate to produce a good student model for style transfer tasks using knowledge distillation.