Semantic Map Based Robot Navigation with Natural Language Input

We present a new semantic map based robot navigation system in the paper. The system takes human voice input, processes multi-modal data including natural languages and RGB-D images, and generates semantic maps for robot navigation. Making use of recent development in image segmentation tools, we integrate robot mapping and localization with a customized real-time object detection model, so that the semantic and navigation layers are efficiently combined for robot navigation purpose. We demonstrate the performance of our developed algorithms in both simulation and real robot experiments. Compared with existing works, we demonstrate applicability to real robot systems and superior performance in terms of success rate.