Closed-Loop Placement Planning for Regrasping and Reconstruction With Single-View RGB-D Images

In robotic reorientation, achieving certain positions or orientations within a workspace can be difficult due to collisions and interference between objects and the environment. Regrasping is a technology that overcomes limitations on object reorientation using multiple pick-and-place actions. However, constructing a regrasping graph, which consists of predicted intermediate placements, is challenging with only partial observations of unknown objects. To address this challenge, we design a closed-loop placement planning system using single-view RGB-D images. This system seamlessly integrates perception and manipulation processes, capturing RGB-D images for object reconstruction while predicting intermediate placements for constructing regrasping graphs simultaneously. To optimize the regrasping graph construction, we propose a Truncated Signed Distance Function (TSDF)-based placement evaluation algorithm that selects the next-best-placement by estimating the information gain and retention of the placements. Additionally, we adopt a novel data fusion method based on predicted intermediate placements to enhance object reconstruction and TSDF updates, effectively mitigating in-hand errors. Our system surpasses baselines in accuracy and diversity of placement predictions. It demonstrates superior performance in object reconstruction, excelling in quantitative analysis and visualization. Real-world experiments validate the effectiveness of our system in reorientation and reconstruction tasks on real industrial parts, demonstrating its robustness and applicability in practical scenarios. Note to Practitioners—This paper was motivated by the problem of object reorientation in robot assembly tasks in the industrial sector. Robots often struggle to reorient objects to the desired orientation through a single pick-and-place due to workspace limitations. Our solution involves capturing RGB-D images of objects and implementing multiple pick-and-place steps, known as regrasping, to reorient the objects effectively. During this process, the point clouds of the objects are reconstructed. A key innovation of our approach is the method for selecting the next-best-placement for each step. This is achieved by estimating the information gain and retention and fusing data to improve the accuracy of both regrasping and reconstruction. Our approach has shown superior performance in both simulated and real-world experiments with industrial parts. Practitioners can use this system to enhance the efficiency and reliability of robotic reorientation tasks. Its practical value lies in its ability to manage complex practical scenarios, making it a valuable tool for automation processes. The system’s dependence on predefined parameters restricts its adaptability to dynamic tasks. Future work will explore the integration of multi-modal language models to improve adaptability and accuracy in handling diverse scenarios.