GraphCortex: A Visual Language Model-Guided Knowledge Graph Based Reasoning Framework for Robotic Long-Term Task Planning

Long-term task planning for robots remains a key focus in robotics research. At present, long-term planning faces challenges such as cumulative errors in sequences and limited scope of semantic modalities in planning. Traditional planning relies on static expert rule databases, which lack flexibility. While knowledge graphs based solely on text can effectively map task relationships, their reasoning capabilities are constrained by issues such as modal uniformity and the delays in updating static knowledge bases. To address these issues, this paper proposes GraphCortex, a “knowledge graph-VLM-agent” tripartite fusion paradigm for long-term task planning. Inspired by the human brain’s superior temporal sulcus, inferior parietal lobe and intraparietal sulcus, GraphCortex integrates visual and motion trajectory data into pure text knowledge graphs, achieving multimodal task nodes and aligning visual, action, and textual semantics. In experiments involving long-term tasks, GraphCortex significantly enhances the effectiveness of longterm capabilities of the imitation learning based on single-demonstration-based paradigm, offering a practical solution to challenges like semantic modality uniformity, fragmented task semantics, and the difficulty of autonomously evolving experiential knowledge in robot long-term task planning.