Bridging Perception and User Needs: Semantic Mapping with Grounded Interpretation using a Dual-Embeddings Approach

Assistive robots must reason beyond object recognition to effectively support human needs within their environments. Understanding not only what is present but also what actions are possible in each location is crucial for enabling context-aware assistance. In this paper, we present a framework for constructing a semantic map that associates spatial regions with both perceptual content and actionable utilities, facilitating the grounding of user needs into relevant areas. Using Multimodal Large Language Models, we extract structured descriptions of objects, their states, and associated utilities from observed scenes. We introduce a dual-embedding retrieval strategy that represents both object-based and utility-based aspects of the environment, enabling the matching of user commands, even when formulated implicitly, to appropriate spatial regions. Experiments in simulated domestic environments demonstrate that our approach enhances user need interpretation and improves the selection of relevant areas compared to baselines based on solely objects, solely utilities, and visual embeddings matching.