Panoptic Perception: A Novel Task and Fine-Grained Dataset for Universal Remote Sensing Image Interpretation

Current remote-sensing interpretation models often focus on a single task, such as detection, segmentation, or caption. However, the task-specific designed models are unattainable to achieve the comprehensive multilevel interpretation of images. The field also lacks support for multitask joint interpretation datasets. In this article, we propose panoptic perception: a novel task and a new fine-grained panoptic perception (FineGrip) dataset to achieve a more thorough and universal interpretation for remote sensing images (RSIs). The new task: 1) integrates pixel-level, instance-level, and image-level information for universal image perception; 2) captures image information from coarse-to-fine granularity, achieving deeper scene understanding and description; and 3) enables various independent tasks to complement and enhance each other through multitask learning. By emphasizing multitask interactions and the consistency of perception results, this task enables the simultaneous processing of fine-grained foreground instance segmentation, background semantic segmentation, and global fine-grained image captioning. Concretely, the FineGrip dataset includes 2649 RSIs, 12054 fine-grained instance segmentation masks belonging to 20 foreground things categories, and 7599 background semantic masks for five stuff classes. Furthermore, we propose a joint optimization-based panoptic perception model. Experimental results on FineGrip demonstrate the feasibility of the panoptic perception task and the beneficial effect of multitask joint optimization on individual tasks. The project page is at: https://ybio.github.io/FineGrip.