Variational Regularization Network With Attentive Deep Prior for Hyperspectral–Multispectral Image Fusion

Hyperspectral–multispectral image (HSI-MSI) fusion relies on a robust degradation model and data prior, where the former describes the degeneration of HSI in the spectral and spatial domains, and the latter reveals the latent statistics of the expected high-resolution (HR) HSI. In practice, the degradation model is often unknown, and the data prior is usually too complicated to be expressed analytically. In this study, we propose a variational network for HSI-MSI fusion (VaFuNet), in which the degradation model and data prior are implicitly represented by a deep learning network and jointly learned from the training data. A variational fusion model regularized by deep prior is first proposed, and then, it is optimized via a half-quadratic splitting and unfolded into a deep network. The deep prior is implicitly represented by a proximity operator. Due to the structural self-similarity, HSI possesses structural recurrences across different scales. To exploit such nonlocal prior and enhance the representability of network, we also propose a multiscale nonlocal attention and embed it into the deep prior proximity. The degradation model and deep prior proximity are jointly learned via end-to-end training. Experimental results on simulated and real-life HSI datasets demonstrate the effectiveness of the proposed VaFuNet HSI-MSI fusion method.