Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction

Hyperspectral pansharpening aims to synthesize a low-resolution hyperspectral image (LR-HSI) with a registered panchromatic (PAN) image to generate an enhanced HSI with high spectral and spatial resolution. Recently, the proposed HS pansharpening methods have obtained remarkable results using deep convolutional networks (ConvNets), which typically consist of three steps: 1) upsampling the LR-HSI; 2) predicting the residual image via a ConvNet; and 3) obtaining the final fused HSI by adding the outputs from first and second steps. Recent methods have leveraged deep image prior (DIP) to upsample the LR-HSI due to its excellent ability to preserve both spatial and spectral information, without learning from large datasets. However, we observed that the quality of upsampled HSIs can be further improved by introducing an additional spatial-domain constraint to the conventional spectral-domain energy function. We define our spatial-domain constraint as the $L_{1}$ distance between the predicted PAN image and the actual PAN image. To estimate the PAN image of the upsampled HSI, we also propose a learnable spectral response function (SRF). Moreover, we noticed that the residual image between the upsampled HSI and the reference HSI mainly consists of edge information and very fine structures. In order to accurately estimate fine information, we propose a novel overcomplete network, called HyperKite, which focuses on learning high-level features by constraining the receptive from increasing in the deep layers. We perform experiments on three semisynthetic and one real HSI datasets to demonstrate the superiority of our DIP-HyperKite over the state-of-the-art pansharpening methods. The deployment codes, pretrained models, and final fusion outputs of our DIP-HyperKite and the methods used for the comparisons will be publicly made available at https://github.com/wgcban/DIP-HyperKite.git.

I. Introduction

Hyperspectral images (HSIs) with a large number of spectral bands have gained immense attention in the field of remote sensing due to its applications in broad research areas such as classification [1], unmixing [2], anomaly detection [3], change detection [4], etc. However, due to the limited incident energy available when capturing an image, hyperspectral imaging systems face tradeoffs between spectral resolution, spatial resolution, and signal-to-noise ratio (SNR) [5]. For this reason, hyperspectral imaging systems can provide images with high spectral resolution but with low spatial resolution. In contrast, multispectral imaging systems can provide data with high spatial resolution but with fewer spectral bands (e.g., panchromatic images or multispectral images (MSIs) with three or four spectral bands). Low spatial resolution in HSIs leads to relatively poor performance in some practical remote sensing applications, such as road topology extraction [6], and spectral unmixing [7]. Therefore, full-resolution HSIs with high spatial and spectral resolution are desired. One way to obtain such ideal HSIs is to fuse high spectral resolution HSIs with high spatial resolution PAN/MSIs. This fusion process is called HS pansharpening in the remote sensing literature, which is indeed a form of super-resolution [8].