Enhancing Vision-and-Language Navigation in Continuous Environment via Data Synthesis

Developed from Vision-and-Language Navigation (VLN), which necessitates the agent to navigate in discrete environments, Vision-and-Language Navigation in Continuous Environment (VLN-CE) poses a challenge in visual navigation due to the intricate task of aligning language with visual cues in continuous settings during navigation. While data synthesis is commonly utilized in VLN to augment datasets, this practice is absent in VLN-CE. Furthermore, existing studies on visual navigation underscore the efficacy of leveraging extensive imitation on a large scale to enhance performance. However, the acquisition of large-scale VLN-CE data proves prohibitively costly. Addressing these challenges, we introduce a methodology for generating multimodal VLN-CE data. Initially, shortest paths are computed based on the start and end positions. Subsequently, the agent is deployed to traverse the path through a sequence of actions. Following this, actions undergo thresholding to identify key actions, from which critical frames are ascertained. Instruction templates are then generated by replacing nouns and actions in the original VLN-CE instructions. New nouns are predicted using a large-scale pre-trained model (CLIP). Finally, these new nouns and actions are inserted into the templates to produce new instructions without human annotation. Experimental findings demonstrate that the performance of VLN-CE is promoted when the generated data scale up.