End-to-end Learning for Encrypted Image Retrieval

Traditional image encryption methods impede the process of extracting features from cipher-images, which makes next step's retrieval become difficult. There are some encrypted image retrieval works that extract features from encrypted images by human initially, then build model to enforce retrieval by these features. In the paper, we propose end-to-end encrypted image retrieval, using deep learning model to extract features from cipher-images and conducting retrieval. We do not need to extract hand-craft features from cipher-images, because our retrieval model can extract features by end-to-end learning. Images are encrypted by block permutation and color value substitution operation for partial blocks. Our retrieval model uses Vision Transformer (ViT) as BackBone, combines triplet loss and cross entropy loss when training. In experiments, we compare different BackBones - ViT and ResNet50, and the results show that the retrieval performance is better when using ViT as BackBone. Our scheme not only achieves end-to-end encrypted image retrieval, but also obtains significant improvement on retrieval performance when compared with current methods.