A Macro-Micro Vision Integrated Micromanipulation System for Self-Initialization and Resilient Control

Robotic micromanipulation systems (RMS) enable precise and repeatable operations under a microscope. Traditional RMS rely solely on microscopic visual feedback, necessitating time-consuming manual positioning to bring the tool tip within the microscope field-of-view (Micro-FOV), which limits efficiency and heavily depends on operator skill. This paper proposes an innovative RMS that integrates macro and micro vision to automate the aforementioned tool tip positioning and facilitate resilient control. The system utilizes an external camera to obtain the macro field-of-view (Macro-FOV), containing the tool and fiducial markers, and estimates the tool tip’s 3D position by triangulation. Visual servoing is then used to guide the tool tip towards the Micro-FOV. Under the Micro-FOV, a tool-sweep detector based on partitioned difference images is used to sequentially locate the tool’s shaft and tip. After auto-focusing, the system executes tool tip and Petri dish resilient control based on our developed self-calibration and self-recalibration mechanisms. During the operation, the system provides an intuitive user interface that includes both macro and micro information, improving the visualization and productivity of micromanipulation. Experiments show that the self-initialization scheme can be implemented across different macro camera viewpoints, reducing the average tip positioning time from 65.70 to 50.08 seconds compared to manual operation, thereby decreasing manual labor intensity and improving efficiency. The self-recalibration mechanism achieves precision and resilient control, with an average error of $0.95~\mu $ m over 25 continuous trials. Additionally, the system exhibits robustness against vibration and visual interference, underscoring its potential for diverse biomedical applications.Note to Practitioners—In the biomedical field, robotic micromanipulation systems (RMS) are valued for their high precision and repeatability. Existing research on RMS typically focuses on achieving varying degrees of automation using visual feedback from the microscope, with the assumption that the tool tip is already within the Micro-FOV prior to the task. However, the initial step of moving the tool tip from the Macro-FOV to the Micro-FOV and eventually bringing it to focus usually requires time-consuming manual operation that highly depends on the skill of individual operators. To achieve automatic positioning and resilient control of the tool tip, this paper presents an RMS that integrates macro and micro vision. The system combines visual feedback from a macro camera and a microscope to complete the self-initialization of the system in three steps: macro tip positioning, micro tip positioning, and self-calibration. After self-initialization, the system provides the operator with an intuitive cursor-based user interface. During operation, the system uses visual feedback to detect the tip position and ensures the control accuracy through a self-recalibration mechanism. The proposed self-initialization method has the potential for a wide range of applications. It can be extended to micromanipulation systems with different types of end-effectors, thereby improving the efficiency and precision of micromanipulation.