Image-Goal Navigation via Metric Mapping and Keypoint based Reinforcement Learning

In this paper, we tackle the problem of Image-Goal Navigation, where an agent is asked to find the viewpoint of a given goal image in an unseen environment. Past methods rely on depth cameras, require panoramic observations and a pre-built graph network, or is trained on limited situations where the image-goal viewpoint is close to the initial agent state. We propose using a hierarchical policy structure which uses a high-level policy to guide the agent more efficiently to the image-goal viewpoint. Our method uses metric maps built by RGB images by Active Neural SLAM network and image descriptors extracted from place recognition networks. We have trained a policy network whose input is a SLAM Map, along with the information about matched keypoints and global descriptors between the observation and goal image, and output is a global goal which guides the agent to the image goal. Our method is trained on a self-made image-goal dataset based on the Gibson datset which contains images of the best viewpoints of semantic objects in the environment and is shown that using this method gives improved performance compared to previous related methods.