Lumipen 2: Dynamic Projection Mapping with Mirror-Based Robust High-Speed Tracking against Illumination Changes

Dynamic projection mapping (DPM) is a type of projection-based augmented reality that aligns projected content with a moving physical object. In order to be able to adjust the projection to fast motions of moving objects, DPM requires high-speed visual feedback. An option to reduce the temporal delay of adjusting the projection to imperceptible levels is to use mirror-based high-speed optical axis controllers. However, using such controllers for capturing visual feedback requires a sufficient amount of illumination of the moving object. This leads to a trade-off between tracking stability and quality of projection content. In this article, we propose a system that combines mirror-based high-speed tracking with using a retroreflective background. The proposed tracking technique observes the silhouette of the target object by episcopic illumination and is robust against illumination changes. It also maintains high-speed, accident-avoidant tracking by performing background subtraction in an active vision system and employing an adaptive windows technique. This allows us to create a DPM with an imperceptible temporal delay, high tracking stability and high visual quality. We analyze the proposed system regarding the visual quality of the retroreflective background, the tracking stability under illumination and disturbance conditions, and the visual consistency relative to delay in the presence of pose estimation. In addition, we demonstrate application scenarios for the proposed DPM system.