A Simple Unknown-Instance-Aware Framework for Open-Set Object Detection

Open-set object detection aims to detect training set objects (known classes) and non-training set objects (unknown classes). The current frameworks rely on human-in-the-loop, i.e., humans provide training data of unknown classes, to be aware of unknown-instance. However, it is difficult to collect unknown-class images in the real world. In this work, we propose a simple unknown-instance-aware framework for open-set object detection, which can extend close-set detectors to open-set object detection. The proposed framework consists of a class-agnostic proposal module and a prompt-guided semantic module. The class-agnostic proposal module generates region proposals, which are used by the prompt-guided semantic module. The prompt-guided semantic module distinguishes unknown instances by the similarity score between embeddings and image features, which is based on the aligned image-text space of pre-trained vision-language models, e.g., CLIP. The experiments show the class-agnostic proposal module is better than the class-aware locator in localization performance, and the prompt-guided semantic module has better unknown-class recognition performance.