Remote Sensing Semantic Segmentation Quality Assessment Based on Vision Language Model

Variations in scene complexity and image quality across remote sensing images lead to inconsistent performance when applying pretrained semantic segmentation models. To ensure quality control for downstream tasks, quality assessment on interpretation results becomes a challenging task, primarily due to the limited availability of reference annotations. To address this issue, we propose RS-SQA, a remote sensing semantic segmentation quality assessment model based on the vision language model (VLM) for images without ground truth. RS-SQA uses a dual-branch architecture that leverages an RS VLM for semantic understanding and utilizes intermediate features from segmentation methods to extract implicit information. We fuse these two feature streams via a semantic-guided module, yielding accurate quality predictions. Specifically, we introduce CLIP-RS, a large-scale pretrained VLM trained on 10 M remote sensing images with purified captions. Feature visualizations confirm that CLIP-RS can effectively differentiate between various levels of segmentation quality. To facilitate the development of RS semantic segmentation quality assessment, we present RS-SQED, a new dedicated dataset sampled from four major RS semantic segmentation datasets and annotated with segmentation accuracy scores from eight representative segmentation methods. Experiments on the established dataset demonstrate that RS-SQA outperforms state-of-the-art (SOTA) quality assessment models and aids in selecting the most suitable pretrained models for remote sensing semantic segmentation by comparing predicted accuracies across different models.