Combining Episodic Memory and LLMs for the Verbalization of Robot Experiences

The ability to communicate past experiences is fundamental for intelligent and natural interaction. Humanoid robots continuously accumulate rich, multi-modal experiential data and must be able to articulate their episodic experiences in natural language to support effective human-robot communication. Existing approaches either rely on the generalization capabilities of large language models (LLMs), sometimes combined with episodic memory, or the precision of rule-based verbalization systems, each presenting limitations when used in isolation. In this work, we present a novel hybrid framework that integrates the adaptability of LLMs with the robustness of rule-based methods and the generalizable structure of memory-based approaches. Our system implements strategies to retrieve and transform memory representations of past perceptions and actions into natural language responses. This enables humanoid robots to respond to natural language queries about their experience. Experimental evaluation based on a set of distinct query types demonstrates that our approach successfully answers 89.4 % of episodic memory questions with human-in-the-loop refinement, while reducing token consumption by 97 % compared to pure LLM-based methods. Furthermore, we demonstrate the system's extensibility by leveraging LLMs, such as GPT-4.1, to expand the range of permissible queries through example-based interaction. The evaluation on our humanoid robot Armar-7 performing household tasks validates that our hybrid approach balances response quality with computational efficiency to address the crucial need for dependable yet flexible verbalization of robot experiences. Code and examples are available at robotexperienceverbalization.github.io/REV.github.io/