A visual/haptic interface to virtual environment (WYSIWYF display) and its application

VR training or skill transfer via virtual environment is a kind of human communication from one (expert) to others (trainee). To build a VR training system for visuo-motor skills, a visual interface should be correctly registered to a haptic interface so that the visual sensation and the haptic sensation are consistent spatially and temporally. Ideally a visual/haptic interface should be configured in such a way that what you see is what you feel as it is in the real life situations. In this paper, a reasonable method to realize correct visual/haptic registration is introduced. This method provides correct visual/haptic registration using a vision-based object tracking technique and a video keying technique. The first prototype called WYSIWYF Display has been built and the proposed concept was demonstrated. The paper then discusses a potential application to VR training using the WYSIWYF Display.