Pansharpening via Super-Resolution Iterative Residual Network With a Cross-Scale Learning Strategy

Pansharpening exploits the high-spatial-resolution panchromatic (HR PAN) images to restore the spatial resolution of the corresponding low-spatial-resolution multi-spectral (LR MS) image, producing a fused image and high-spatial-resolution multi-spectral (HR MS) image. Recently, many methods based on convolutional neural networks (CNNs) have been put forth for the pansharpening task, but most of them still have some limitations, such as the simple stacked convolutional architectures resulting in information distortion, and some scale-related problems caused by the supervised learning strategy. Therefore, we propose a method named super-resolution iterative residual (SRIR) network with a cross-scale (CS) learning strategy to overcome these drawbacks. Regarding the SRIR we propose, we design an upsampling network based on a sub-pixel convolution structure to replace the traditional upsampling pre-processing. We adopt the iterative networks framework and design a new spatial information injection module to continuously inject spatial and spectral features into the network, which can enhance the information flow and transmission. We produce approximate HR MS with a guidance filter and map the residual information between the approximate HR MS and the reference HR MS by SRIR to enhance the quality of fused images. Regarding the CS we propose, we train the network at degraded scale, which is named deep prior, and then design a finer-scale unsupervised fine-tuning loss function to refine the network parameters with deep priors, to overcome the scale effect. Experiments show the following: 1) SRIR-based pansharpening method can obtain the best result at the degraded scale; 2) the scale-effect is negatively correlated with the depth of the network, meaning that the deeper the network, the stronger the robustness to scale effect; 3) the CS learning strategy can widely improve the performance of CNNs-based pansharpening methods in full-resolution; and 4) our method can produce better results at full-resolution scale than all the other traditional and deep learning methods.