U²ConvFormer: Marrying and Evolving Nested U-Net and Scale-Aware Transformer for Hyperspectral Image Classification

Hyperspectral image (HSI) classification plays an important role in the human exploration of the Earth. Recent research of deep learning-based HSI classification has been fast-growing, but still suffers from three obstacles: First, existing deep learning-based HSI works lack of extraction and utilization of multigrained multiscale information and multiscale local-to-global information. Second, most previous works have too fixed-sized receptive fields in their convolutional network parts to handle HSI classification problems, and pay no attention to the existence of asymmetries in the spectral-spatial dimension of the HSI data. Third, most networks for HSI classification are hand-craft. To this end, we propose a novel architecture in this article, which is the first to combine the advantages of nested U-Net and scale-aware Transformer, named U2ConvFormer. Specifically, the nested U-Net structure can fully extract and aggregate multiscale spectral-spatial features at both inter- and inner stage granularity. The scale-aware Transformer takes multiscale local spectral-spatial features from the encoder of nested U-Net and produces multiscale global spectral-spatial features for its decoder. After that, we design a novel plug-and-play searchable operation called asymmetric spectral-spatial convolution (A2SConv), where asymmetric spectral-spatial feature pooling and multiscale feature extraction can be concurrently searched. Furthermore, we develop a customized search strategy to automatically design U2ConvFormer, which uses advanced neural architecture search (NAS) methods to enable the customization of suitable models for different hyperspectral datasets. Experimental results on three benchmark datasets, including Indian Pines, Pavia University and Houston University 2018, validate the superiority of our proposed U2ConvFormer, which achieves new state-of-the-art performance across different benchmark datasets.