A multimodal augmented reality DJ music system

One of the goals of human-computer interaction is to utilize more intuitive and natural methods for communication. This goal has become of particular importance in augmented reality (AR) applications. Developing such interaction techniques is critical since traditional interfaces (keyboard, mouse) are cumbersome and awkward in AR environments. This paper provides a description of a real-time interactive multimodal augmented reality entertainment system. The system design is based on the DJ music environment and consists of four main modules, namely a speech recognition module, a head tracking and graphic engine module, a gesture tracking module and finally, a sound synthesis module. Using our system, the user can scratch the music through gesture movement as if using a DJ turn table. The user can also enjoy real time interaction with virtual 3D dancers augmented on the table through voice commands and hand gestures.