Integrating grasp planning and visual feedback for reliable manipulation

We present a vision-centric manipulation framework for reliably performing reach-and-grasp tasks in everyday environments. By combining grasp planning and visual feedback algorithms, and constantly considering sensor visibility, the framework can recover from sensor calibration errors and unexpected changes in the environment. Although many current robot systems include planning and vision components, these components are treated independently, which reduces the capability of the system from making informed decisions. Our proposed framework incorporates information in a data-driven way from both planning and vision modalities during the planning and execution phases of the task. The planning phase generates a plan to move the robot manipulator as close as safely possible to the target object such that the target is easily detectable by the on-board sensors. The execution phase is responsible for continuously choosing and validating a grasp for the target while updating the environment with more accurate information. We stress the importance of performing grasp selection for the target during visual-feedback execution because more precise information about the target's location and its surroundings is available. We evaluate our framework on several robot platforms in simulation.