Unsupervised Pansharpening Based on Self-Attention Mechanism

Pansharpening is to fuse a multispectral image (MSI) of low-spatial-resolution (LR) but rich spectral characteristics with a panchromatic image (PAN) of high spatial resolution (HR) but poor spectral characteristics. Traditional methods usually inject the extracted high-frequency details from PAN into the upsampled MSI. Recent deep learning endeavors are mostly supervised assuming that the HR MSI is available, which is unrealistic especially for satellite images. Nonetheless, these methods could not fully exploit the rich spectral characteristics in the MSI. Due to the wide existence of mixed pixels in satellite images where each pixel tends to cover more than one constituent material, pansharpening at the subpixel level becomes essential. In this article, we propose an unsupervised pansharpening (UP) method in a deep-learning framework to address the abovementioned challenges based on the self-attention mechanism (SAM), referred to as UP-SAM. The contribution of this article is threefold. First, the SAM is proposed where the spatial varying detail extraction and injection functions are estimated according to the attention representations indicating spectral characteristics of the MSI with subpixel accuracy. Second, such attention representations are derived from mixed pixels with the proposed stacked attention network powered with a stick-breaking structure to meet the physical constraints of mixed pixel formulations. Third, the detail extraction and injection functions are spatial varying based on the attention representations, which largely improves the reconstruction accuracy. Extensive experimental results demonstrate that the proposed approach is able to reconstruct sharper MSI of different types, with more details and less spectral distortion compared with the state-of-the-art.

I. Introduction

Multispectral image (MSI) is one of the most widely utilized satellite optical images. It usually covers the electromagnetic spectrum at the visible and near-infrared wavelengths with up to eight spectral bands. Due to the rich spectral characteristics, MSI has been deeply involved in various human activities, including, but not limited to, environmental change detection [1], agriculture monitoring [2], [3], and weather forecasting [4]. Very often, the analysis results of these applications rely heavily on both the spatial and spectral resolution of MSI. However, due to the physical limitations of satellite optical sensors, the high-spectral resolution of MSI can only be achieved by sacrificing its spatial resolution. On the other hand, satellite sensors are also able to acquire panchromatic images (PAN) with high spatial resolution although the spectral resolution is poor with only one spectral band.