Look into Gradients: Learning Compact Hash Codes for Out-of-distribution Retrieval

Hashing aims to compress raw data into compact binary descriptors, which has drawn increasing interest for efficient large-scale image retrieval. Current deep hashing often employs evaluation protocols where usually query data and training data are from similar distributions. However, more realistic evaluations should take into account a broad spectrum of distribution shifts with varying degrees. Therefore, we study the problem of out-of-distribution generalization in image retrieval, which seeks to learn a retrieval model from a source domain and generalize to unseen target domains. However, this problem is challenging owing to data scarcity in target domains and the potential overfitting of domain-specific patterns. Here, we propose a novel hashing model named Looking-into-gradients (LOG) for image retrieval under out-of-distribution shifts, which comprehensively explores gradients for both data generation and model optimization. Specifically, to overcome data deficiency in target domains, we formalize the worst-case problem to generate challenging virtue samples via adversarial gradient ascend. Besides, to further enhance model generalization capability, we not only identify non-crucial parameters with minor gradients and values and shrink them to zero, but also modify the inconsistent gradients across domains to prevent learning domain-specific patterns. Extensive experiments on various datasets demonstrate that LOG outperforms state-of-the-art methods by up to 8.54%.