A Unified Pansharpening Method With Structure Tensor Driven Spatial Consistency and Deep Plug-and-Play Priors

Pansharpening is to generate a high-resolution multispectral (HRMS) image by preserving the spectral information from a low-resolution multispectral (LRMS) image and the spatial content from a panchromatic (PAN) image. This article proposes a unified pansharpening method with structure tensor driven spatial consistency and deep plug-and-play priors. First, the spectral fidelity constraint between HRMS and LRMS is imposed for preserving spectral information. Second, the structure tensor is applied to characterize the spatial geometric information of HRMS and PAN images, thus the structure tensor driven spatial consistency prior between HRMS and PAN is particularly exploited for preserving spatial content. Moreover, by generalizing the convolution neural network (CNN) fusion method into a unified variational framework, a novel CNN-based deep plug-and-play prior between the HRMS and CNN-based fused multispectral (MS) images is also proposed to generate more image characteristics for further preserving spectral information and spatial content. Besides, the proposed model is solved by the alternating direction method of multipliers (ADMMs) algorithm. Finally, extensive experiments on both reduced and full resolution by comparing with various representative approaches exhibit the excellent performance of the proposed method.