Cross-Modal Graph Knowledge Representation and Distillation Learning for Land Cover Classification

Complementary multimodal remote sensing (RS) data often lead to more robust and accurate classification performance. However, not all modal data can be available at the time of inference due to imaging conditions. To mitigate this issue, cross-modal knowledge distillation becomes an effective method, as it can leverage the complementary characteristics of multimodal data to guide cross-modal classification in cases with missing data. Therefore, this article examines the shortcomings of traditional convolutional neural network (CNN) cross-modal distillation methods in land cover classification: 1) insufficient knowledge representation and 2) unstable knowledge transfer. Moreover, a novel cross-modal graph knowledge representation and distillation learning (CGKR-DL) framework is proposed to enhance land cover classification performance. The proposed CGKR-DL designs a single-stream joint feature learning network with CNN and the graph convolutional network (GCN) to effectively construct the remote topology of data based on the strong correlation between land objects, thus enhancing the knowledge representation ability of the network. In addition, a multigranularity graph distillation method is proposed to compensate for the inability of traditional CNN distillation in handling graph-structured information, where a feature distillation module based on graph discrimination (FD-GDM) is designed for stable graph feature distillation. We evaluate CGKR-DL on three publicly available multimodal RS datasets [hyperspectral (HS)-light detection and ranging (LiDAR), HS-synthetic aperture radar (SAR), and HS-SAR-digital surface model (DSM)] and achieve a significant improvement in comparison with several state-of-the-art methods.