Multimodal Task Attention Residual Reinforcement Learning: Advancing Robotic Assembly in Unstructured Environment

Robotic assembly in dynamic and unstructured environments poses challenges for recent methods, due to background noise and wide-ranging errors. Directly learning from environments relies on complex models and extensive training iterations to adapt. Representation selection approaches, which depend on expert knowledge, can reduce training costs but suffer from poor robustness and high manual costs, limiting scalability. In response, this letter proposes a system that integrates task attention into residual reinforcement learning to address these challenges. By effectively segmenting task-relevant information from the background to leverage task attention, our approach mitigates the impact of environmental variability. Additionally, compared with existing baselines, our task attention mechanism based on instance segmentation and prompt-guided selection does not require additional offline training or local fine-tuning. Experimental evaluations conducted in both simulated and real environments demonstrate the superiority of our method over various baselines. Specifically, our system achieves high efficiency and effectiveness in learning and executing assembly tasks in dynamic and unstructured environments.