Lightweight High-Resolution Remote Sensing Scene Classification via Adaptive Enhanced Knowledge Distillation

Scene classification is a key step in intelligent information processing of high-resolution remote sensing, which aims to identify the land-use types of remote sensing image blocks. In recent years, the models based on deep convolution neural networks (CNN) have made outstanding achievements in the field of remote sensing, but these models are computationally expensive and time-consuming, while the lightweight shallow network models have few parameters, fast speed, but low accuracy. Both the deep CNN models and the lightweight shallow network models could not be directly applied to embedded devices. Therefore, we propose an Adaptive Enhanced Knowledge Distillation (AE-KD) to deeply mine the output and feature information from teacher model and transfer them to student model, so as to improve the performance of lightweight model. Firstly, aiming at the uneven degree of difference among remote sensing image categories, an adaptive temperature mechanism is proposed by improving the temperature mechanism in the traditional knowledge distillation, which promotes the student model to better learn the probability distribution knowledge from the output layer of the large and deep teacher model. And then, the spatial attention and inter-channel correlation of features are added as constraints in order to make the student model learn the multi-level knowledge from teacher model. The experimental results on UC Merced Land-Use and AID public datasets show that the proposed method reduces 91 % parameters of the teacher model and improves the prediction speed by 22 times, where it has only a small loss of classification accuracy, which is effective for the lightweight model. ablation study also further analyzes and discusses the performance improvement of the student model under different levels of knowledge distillation.