HabitatDyn 2.0: Dataset for Spatial Anticipation and Dynamic Object Localization

The ability of a robot to perceive and understand its environment is crucial for its actions and behavior. Humans are adept at using semantic information for object localization and path planning, a skill that robots need to emulate for intelligent adaptation in dynamic settings. Training of the spatial anticipation ability, which can enhance spatial perception through semantic understanding, necessitates the availability of appropriate data. Although extensive research has been conducted on datasets for outdoor environments, especially in the context of autonomous driving, there is still a notable lack of datasets specifically designed for indoor environments, with a focus on dynamic object localization. This paper introduces HabitatDyn 2.0, a dataset specifically designed for enhancing object localization capabilities with semantic information from a robot’s perspective. Besides RGB videos, semantic annotations, and depth information, HabitatDyn 2.0 also features top-down view labels for dynamic objects, which is required for training the spatial anticipation ability based on semantic information. Additionally, an algorithm that leverages spatial anticipation for dynamic object localization is presented, trained, and evaluated on the dataset.