Toward Open-World Semantic Segmentation of Remote Sensing Images

In this work, we address the challenge of open-world semantic segmentation for remote sensing (RS) images, which involves segmenting arbitrary objects in images using open RS data. Previous efforts in open-world segmentation mostly focus on Internet-scale paired image-text data with rich vocabulary of concepts. However, these works cannot be directly transferred to RS domain due to the lack of large-scale RS data-text pairs and the corresponding annotations. To overcome this limitation, we propose using text descriptions and annotations from OpenStreetMap as a source of supervision while using images from satellite images. We utilize a conditional Unet model to predict segmentation masks given a text description, and leverage the rich information contained in a pretrained CLIP model to align the images and the corresponding text embeddings using a contrastive loss. Our experimental results demonstrate the potential of open-world segmentation on open RS data.