TIP: Tangible Interactive Projector with Projection Touch Tracking and Spatial Augmented Reality

Projection interaction is a new method of human-computer interaction which creates viewable screen in the actual space and can be directly touched and interacted by users, which can be universally used in the field of robotics and consumer electronics. In this work, we propose a new device named tangible interactive projector to effectively enable projection touch tracking and spatial augmented reality, in which we use object detection, instance segmentation and keypoints detection to improve the efficiency of touch detection and reduce the probability of misrecognition. We implement the spatial augmented reality method with the combination of the projection unit, computing unit, and perception unit. The perception unit consists of a high definition camera and an infrared camera, providing more modal data for touch recognition, which also makes it possible to display the projection adaptively according to the type and shape of the object in the field view using the bounding box output by object detection. The experiments show that our proposed equipment and interactive method perform significantly better than existing equipment or methods.