Unleashing Knowledge Potential of Source Hypothesis for Source-Free Domain Adaptation

Source-Free Domain Adaptation (SFDA) task aims to transfer knowledge from a labeled source domain to a label-scarce target domain, in which the source data can not be accessed but only a pre-trained source model and unlabeled target data are available during adaptation. Previous methods for source model adaptation rely on hypothesis transfer learning that trains the feature extractor to learn target features aligned to the distribution of source features while freezing the source classifier. However, reusing only the source classifier without exploring the comprehensive knowledge of the source model can lead to biased feature alignment. To this end, we propose a novel method called Transformer-bAsed thorouGh Source HypOthesis Transfer (TagSHOT) framework to effectively unleash the thorough knowledge potential of pre-trained source hypothesis. Specifically, our approach delves into the correlation coefficient among CLS/patch tokens across different Transformer layers, uncovering the concealed insights within the pre-trained source model and constructing a comprehensive source hypothesis. By tailoring the target feature alignment to this thorough source hypothesis, our model facilitates the adaptation of a broader range of classification-related knowledge to the target domain. Furthermore, we introduce a Salient Token Extension (STE) module, designed to capture the target-specific discriminative information by propagating the salient information among tokens. This mechanism enriches our model's ability to understand and incorporate target-specific nuances. Extensive experiments have been conducted to validate the effectiveness of our method, which outperforms state-of-the-art approaches by a large margin.