Attention map feature fusion network for Zero-Shot Sketch-based Image Retrieval

Zero-shot sketch-based image retrieval (ZS-SBIR) is a great and important computer vision problem. The image category in the test phase is a new category that was not visible in the training stage. Because sketches are extremely abstract, the commonly used backbone networks (such as VGG-16 and ResNet-50) cannot handle both sketches and photos. To solve this problem, we propose a novel and effective feature embedding model called Attention Map Feature Fusion (AMFF). The AMFF model combines the excellent feature extraction capability of the ResNet-50 network with the excellent representation capability of the attention network. By processing the residuals of the Res Net-50 network, the attention map is finally obtained without intro-ducing external semantic knowledge. Most previous approaches treat the ZS-SBIR problem as a classification problem, which ignores the huge domain gap between sketches and photos. This paper proposes an effective method to optimize the entire network, called domain-aware triplets (DAT). Domain feature discrimination and semantic feature embedding can be learned through DAT. In this paper, we also use the classification loss function, which is used to stabilize the training process to avoid getting trapped in a local optimum. Our code and related datasets are publicly available at https://github.com/haizhu12/ammln.git.