Viewpoint-Dependent 3D Visual Grounding for Mobile Robots

3D visual grounding is the task of identifying objects in spatial environments based on textual descriptions, enabling natural language interactions between humans and robots. However, existing studies overlook viewpoint-dependent texts expressions, such as "the chair to your right", despite their frequent use in human instructions. In this paper, we introduce a novel problem setting focused on viewpoint-dependent texts and present a new dataset that incorporates the robot’s viewpoint. We conducted three experiments to analyze the dataset’s difficulty and characteristics by comparing existing models and newly designed models that take viewpoint as an additional input. Our results indicate that considering viewpoint is crucial for the object selection process in our task. In addition, we also found that the difficulty of the task varies depending on how the object is described in the text.