A Prototypical Metric Learning Approach for Open-Set Semantic Segmentation on Remote Sensing Images

Semantic segmentation has received wide attention as a feasible solution to effectively interpret the information in remote sensing images. Solutions are typically built with a static closed-set perception, where all labels are known a priori. However, in real-world applications, such as remote sensing images, one has to handle objects from unknown classes. Open-set semantic segmentation (OSSS) is an approach that incorporates open-set perception into semantic segmentation, allowing the recognition of unknown classes of objects. Different studies have explored the use of OSSS in remote sensing images. However, their performance is limited due to the poor and overlapped representation of the features extracted from images. This results in an embedding space with low discrimination among the classes. This article introduces a novel loss function called prototypical triplet loss, which uses prototype representation and metric learning to improve open-set recognition. In addition, two open-set classifiers, one based on principal components and the other on prototypical distance, were also proposed once they took advantage of the features obtained by the prototypical triplet loss. Experiments were done with two public remote sensing image datasets: Vaihingen and Potsdam. The results demonstrate that the proposed methods improve OSSS compared to other state-of-the-art approaches. These results reinforce the importance of this type of approach, enabling applications in real systems that require open-set recognition. All codes are freely available at
https://github.com/Brilhador/tgrs2023
to foster further research in this area.