PromptSeg: Prompt for Universal Remote Sensing Semantic Segmentation

Semantic segmentation in remote sensing (RS) involves classifying each pixel of an image into predefined categories. Despite existing deep learning-based methods having significantly advanced semantic segmentation performance, two key challenges remain: 1) these methods are task-specific and exhibit limited generalization across different RS scenario tasks and 2) they require prior knowledge of the task types to select the appropriate model. To address the above challenges, we propose PromptSeg, a prompt learning method for universal RS semantic segmentation based on the segment anything model (SAM). Specifically: 1) to achieve efficient adaptation to diverse RS tasks with minimal fine-tuning, PromptSeg leverages the unparalleled generalization capabilities of SAM and 2) we propose task-specific prompts that encode essential image features and key distinctions to reduce the reliance on prior knowledge. These task-specific prompts consist of two implicit prompts and one explicit prompt (E-Prompt): the task-related implicit prompt (TI-Prompt), which learns task scenario types; the feature-aware implicit prompt (FI-Prompt), which captures essential image features; and the E-Prompt, which facilitates accurate boundary identification and delineation. To effectively integrate these prompts, we introduce the prompt learning module (PLM), which coordinates their fusion to enhance segmentation performance. Extensive experiments demonstrate the superior performance of PromptSeg across various scenarios, including road extraction, field extraction, cloud segmentation, and building segmentation.