Learning Deep Frequency Degradation Prior for Remote Sensing Spatio-temporal Fusion

Existing deep learning-based remote sensing spatiotemporal fusion (STF) relies on a data-driven paradigm without considering the degradation prior modeling from the coarseto fine-resolution images. This makes the learned model easy to overfit to the training dataset, resulting in poor domain generalization across different datasets. To this end, this paper presents a deep frequency degradation prior to STF, dubbed as DeepFDP. The DeepFDP is based on a statistical observation that the frequency feature distributions of the tokens from the coarse- and fine-resolution images in different datasets have a low intra-resolution variance and a high inter-resolution variance with compact well-separated clusters. Therefore, the DeepFDP first designs a frequency proximity module to learn the mapping function between the token frequency representations of the coarse- and fine-resolution images, which is to narrow their feature distribution difference. Since the statistical properties of the token frequency representations are independent of the land-cover classes, the DeepFDP can be learned on a training set of limited image pairs without extra-supervision signals, which has a favorable zero-shot generalization capability across different datasets. Then, to faithfully recover the land-surface details, a high-frequency feature modulation module is designed that uses the fine-resolution image as guidance to progressively learn the multi-scale residual features in a coarse-to-fine fashion, yielding the fused features with rich high-frequency details. Finally, the progressively-fused features at each stage are fed into a hybrid fusion module, yielding the fine-resolution image prediction. Extensive evaluations on LGC and CIA datasets demonstrate favorable performance of the DeepFDP over state-of-the-art methods. Especially, the DeepFDP also shows a good zero-shot generalization performance when training on LGC and testing on CIA.