Boosting Fashion Image Attributes Classification Performance with MT-GAN Training Technique

Automatic understanding of the product images, particularly the image attributes such as pattern, color, category, material, department, etc., has become a prerequisite for many downstream applications in the e-commerce ecosystem such as visually similar product retrieval and recommendation, publisher website image content monetization. With the recent advancement of deep learning techniques, multi-task deep neural networks (MTDNN) have been widely adopted for multi-class multi-label fashion image classification tasks. In this work, we propose a training technique Multi-Task Generative Adversarial Network (MT-GAN) to improve fashion image classification performance with an additional image translation task. We evaluated the proposed technique on two real-world fashion image datasets and experimental results show that both image classification and image translation benefit from this joint training setting. Specifically the image classifier, even though trained with a smaller amount of data and fewer types of labeled information, manages to outperform existing image classification models such as WTBI and DARN by 54% and 20% respectively for category classification on DeepFashion benchmark. Two main contributors to the superior performance of the image classifier trained as the discriminator in the proposed MT-GAN framework are: 1) the classifier leverages additional amount of data and the regularization effect by taking on the task of a traditional discriminator. 2) The classifier takes not only the RGB image but also the conditional input image shared between the generator and itself as its input, boosting the classification accuracy and recall by 18%. The proposed MT-GAN training technique is GAN-agnostic and can be applied to various state-of-the-art designs of generator and discriminator as well.