Transformer-Driven Semantic-Spatial Adaptive Fusion Representation for Object-Goal Navigation

Visual object-goal navigation requires an agent to make decisions to search for specified target objects within an unknown environment. While learning-based approaches have achieved progress, they still face two limitations: 1) visual representation: the lack of a low-dimensional representation that balances object priors with current observations, and the absence of effective modeling of scene spatial structures, which restricts the agent’s ability to perceive its surroundings. 2) Policy learning: existing reward functions often fail to consider the visual essence of object goal-driven tasks, leading to inefficient learning and suboptimal performance. To address these issues, this paper introduces a semantic-spatial fusion representation framework that incorporates both object semantics and the intrinsic spatial structure of the scene. Specifically, a new object context matrix captures semantic relationships between objects while providing distinct low-dimensional representations for different observations, and an episode memory graph is also constructed and dynamically updated based on observation similarity and geodesic distance to represent the spatial structure of the agent’s environment in real-time. Then, a transformer-based adaptive multimodal feature fusion module is proposed to integrate these dual representations. Moreover, a sparse reward function is designed based on the target’s bounding box to guide the agent to learn correctly. The proposed method is the first to be evaluated on both the simulation platform AI2-THOR and the real-world dataset AVD, demonstrating its generalization in unseen environments. The method is also deployed on a physical mobile robot and tested in real-world scenarios, further validating its practical effectiveness. Note to Practitioners—This work aims to enhance the visual object-goal navigation capabilities of mobile robots in unknown environments. To achieve this, we propose a dual visual representation that incorporates an enhanced object context matrix and an episode memory graph, enabling an agent to recognize object semantics in the current view while perceiving the spatial structure of its surroundings. These features are adaptively fused using a multimodal feature fusion module based on a transformer with a variant of attention mechanism. Additionally, a sparse reward function is designed based on the bounding box of the target object in the current observation to better guide the model training. Extensive experiments demonstrate that, benefiting from the unique visual representation and the newly designed sparse reward, the proposed method allows agents to locate target objects more quickly and accurately. It also enables agents to terminate tasks autonomously at a more appropriate position, ensuring closer proximity to the target and optimal observation views. Throughout the navigation process, the method can flexibly avoid obstacles in its path, particularly when entering narrow spaces, and prevents task failure due to the lack of perception of the surrounding structure. However, our experiments are currently limited to small-scale home environments. Future research will extend the testing to more complex and diverse scenarios, such as restaurants, hospitals, and factories, to further evaluate the method’s robustness and adaptability in real-world applications.