Self-initialization and recovery for uninterrupted tracking in vision-guided micromanipulation

In this paper, we propose a workflow algorithm for timely tracking of the tool tip during cell manipulation using a template-based approach augmented with low level feature detection. Doing so addresses the problem of adverse influences on template-based tracking during tool-cell interaction while maintaining an efficient track-servo framework. This consideration is important in developing autonomous robotic vision-guided micromanipulators. Our method facilitates vision-guided micromanipulation autonomously without manual interventions even during tool-cell interaction. This is done by decomposing the process to four scenarios that operate on their respective mode. The self-initializing mode is first used to localize and focus a region of interest (ROI) which the tip lies in. Once in focus, the tip is manipulated using a unified visual track-servo template-based approach. A reinitialization mechanism will be triggered to prevent tracking from being interrupted by partial cell occlusion of the tracking ROI. This mechanism uses the self-initializing concept combining motion cue and low-level feature detection to localize the needle tip. Following the reinitialization, we further recover tracking of the needle tip using a mechanism that updates the base template. This adaptive approach ensures uninterrupted tracking even when the cell is interacting with the tool and under deformation. Results demonstrated that with the newly incorporated mechanisms, the localized position improved from an error of more than 50% to less than 10% of the specimen size. When there is no specimen in the scene the new workflow shows no adverse effect on the localization through 270 tracked frames. By incorporating reinitialization and recovery to this workflow algorithm, we hope to initiate the first step towards uncalibrated autonomous vision-guided micromanipulation process.