DSNet: Dynamic Stitchable Neural Network for Hyperspectral Image Classification

Hyperspectral image classification (HSIC) aims to identify land-cover categories by leveraging the spectral and spatial information contained in hyperspectral images (HSIs). Currently, many deep learning approaches utilize dual-branch networks to process spectral and spatial data separately, followed by the application of specialized modules to facilitate feature interaction or fusion. However, the design of these modules demands considerable time and effort from researchers and may not adequately capture the inherent relationships between independent spatial and spectral features dynamically. To address these issues, we propose the dynamic stitchable neural network (DSNet) for HSIC. While the DSNet maintains a dual-branch structure, it operates without traditional feature fusion or interaction. Instead, it employs a stitching network approach to integrate the two branches. Specifically, a spatial–spectral stitching module is presented to incorporate multiple stitching layers at various positions between the two network branches, creating new stitched networks that retain the strengths of both original networks. Additionally, a reinforcement learning-based strategy (RLBS) is designed for dynamically selecting stitching positions tailored to specific datasets, enabling the model to adaptively optimize the integration of spatial and spectral features. Recognizing the effectiveness of the Vision Transformer (ViT) in learning spatial information and the capability of a 1-D convolutional neural network (1DCNN) in capturing spectral details, the DSNet directly stitches these two networks together. This fusion maximizes the utilization of both foundational networks, yielding a new hybrid network that delivers exceptional performance while also alleviating the burden on researchers to develop new architectures from scratch. Extensive experiments and analyses conducted on three public HSI datasets demonstrate the superiority of the proposed method, validating the effectiveness of our innovative modules. The codes of this work will be available at https://github.com/ZZC/IEEE-TGRS-DSNet