Robot Navigation Based on 3D Scene Graphs with the LLM Tooling*

This contribution addresses the integration of 3D Scene Graphs (3DSGs) and Large Language Models (LLMs) to advance robotic navigation. We explore how to best prepare 3DSGs for navigation and examine how key features of 3DSGs affect the way LLMs interpret the information encoded in the 3DSGs and create instructions from it. To improve the reliability and accuracy of the generated instructions for navigation, we introduce prompting methods and scoring mechanisms that reduce uncertainty in LLM responses and ensure the validity of the task planning results. In addition, we focus on difficult navigation task queries involving negation and numbers, analyzing how this approach combining 3DSGs with LLMs behaves and devising strategies that lead to an enhanced performance. These findings and results accomplish our goal of contributing to the development of more robust and intelligent robotic navigation systems that effectively leverage the power of 3DSGs and LLMs.