Relationship-Aware Unknown Object Detection for Open-Set Scene Graph Generation

Scene graph generation (SGG) aims to detect the relationships of objects in an image. Recently, it has been extended to open-set SGG, which also considers unknown objects unseen in a training phase and thereby enables various applications in complex real-world scenes. However, previous research on open-set SGG addressed unknown object detection simply by thresholding confidence scores from object classification trained only for known objects. In reality, these scores become low for both unknown objects and failure detections of the background since they look different from known objects. Therefore, the current state of the art of open-set SGG cannot distinguish unknown objects from backgrounds, thereby overlooking their relationships. In this paper, we propose a novel relationship-aware unknown detection technique. Our main idea is to exploit the fact that only foreground regions containing objects can have relationships with other regions. To this end, we define a Bayesian model on objects and relationships and derive an algorithm of variational inference, which propagates foregroundness between regions and region pairs to assign foreground regions that have more related objects and relationships. As the results of extensive experiments using a public benchmark for open-set SGG, the proposed technique outperformed previous methods, including the state-of-the-art thresholding technique, in the standard OSGDet metrics regardless of the SGG models with which the proposed technique was combined (e.g., +0.61 improvement in OSGDet@100 with the VCTree model).