Poster: Physically-based natural hand and tangible AR interaction for face-to-face collaboration on a tabletop

In this paper, we present an AR framework that allows natural hand and tangible AR interaction for physically-based interaction and environment awareness to support face-to-face collaboration using Microsoft Kinect. Our framework comprises of six major components: (1) marker tracking (2) depth acquisition (3) image processing (4) physics simulation (5) communication and (6) rendering. The resulting augmented environment supports occlusion, shadows, and physically-based interaction of real and virtual objects. We propose three methods of natural hand representations including mesh-based, direct sphere substitution and variational optical flow. A tabletop racing game and AR Sandbox applications are created based on this framework, showing the application possibilities.