Towards Open-Set Scene Graph Generation With Unknown Objects

Scene graph generation (SGG) aims to detect objects and their relationships in an image, thereby enabling a detailed understanding of a complex scene for various real-world applications. In SGG applications such as robot vision, it is important to correctly detect all objects without recognizing any object as another kind of object or ignoring it. However, previous studies on SGG do not consider unknown objects whose classes are unseen in training. Consequently, current SGG methods wrongly classify them as known object classes or overlook them. In this paper, we propose a new problem named “open-set SGG” with unknown objects, focusing on detecting even unknown objects and their relationships. Specifically, we formally define this new problem and propose an evaluation protocol, including an extended dataset with unknown objects and novel evaluation metrics designed for the open-set setting. We also build baseline methods by employing and extending existing SGG methods and compare them through experiments to establish the current baseline performance of open-set SGG. Finally, we discuss the limitations of the current SGG methodology in the open-set setting and point out future research directions.