Embodied Reasoning with Self-Feedback

Large Language Models (LLMs) based on Transformer architecture have achieved remarkable success in Natural Language Processing (NLP). However, translating open-ended instructions into granular action plans for robotic systems remains a challenge. This paper proposes a self-supervised learning framework that enables an embodied multi-modal agent to comprehensively understand and perform tasks based on complex natural language interactions. The methodology focuses on an agentic workflow and self-refinement process, where the LLM actively engages in the task-solving process by generating its own problems, decomposing them, and refining the solutions through iterative self-feedback. The proposed methodology is evaluated on a maze navigation task, where an agent must extract sub-tasks from natural language instructions and navigate a grid while satisfying constraints. Experimental results demonstrate the effectiveness of the proposed approach in solving problems with multiple sub-tasks. Such capabilities could transform human-computer interaction and automated decision support.