Adapting Segment Anything Model to Aerial Land Cover Classification With Low-Rank Adaptation

Recently, vision foundation models have gathered wide attention. Among the many endeavors, the segment anything model (SAM) makes remarkable progress toward a universal model showing unprecedented generalization ability. We propose a novel semantic segmentation model that combines SAM’s image encoder and a low-rank adaptation (LoRA) approach for feature extraction and fine-tuning on aerial images. We also employ an auxiliary CNN encoder to facilitate downstream adaptation and complement the ViT encoder on dense vision tasks. Furthermore, cross-attention is utilized to implement feature interactions between the two encoders. Finally, the UperNet head is employed for multiscale feature fusion and generating segmentation masks. The proposed model was evaluated on the ISPRS Vaihingen and Potsdam datasets and achieved the best mean intersection-over-union (mIoU) of 76.44 and 78.01 for the two datasets. The evaluation results demonstrate the superiority of our model.