Adapting Segment Anything Model to Aerial Land Cover Classification with Low Rank Adaptation

Recently, vision foundation models have gathered wide attention. Among the many endeavors, the Segment Anything Model (SAM) makes remarkable progress toward an universal model showing unprecedented generalization ability. We propose a novel semantic segmentation model that combines SAM’s image encoder and a low rank adaptation approach (LoRA) for feature extraction and fine-tuning on aerial images. We also employ an auxiliary CNN encoder to facilitate downstream adaptation and complement for the ViT encoder on dense vision tasks. Further, cross-attention is utilized to implement feature interactions between the two encoders. Finally, the UperNet head is employed for multi-scale feature fusion and generating segmentation masks. The proposed model was evaluated on the ISPRS Vaihingen and Potsdam dataset, and achieved the best mean intersection-over-union (mIoU) of 76.44 and 78.01 for the two datasets. The evaluation results demonstrate the superiority of our model.