Using eye gaze, head pose, and facial expression for personalized non-player character interaction

True immersion of a user within a game is only possible when the world simulated looks and behaves as close to reality as possible. This implies that the game must ascertain, among other things, the user's focus and his/her attitude towards the object or person focused on. As part of the effort to achieve this goal, we propose an eye gaze, head pose, and facial expression system for use in real-time games. Both the eye gaze and head pose components utilize underlying 3D models, while the expression recognition module uses the effective but efficient LBP-TOP approach. We then demonstrate the utility of this system in a test application wherein the user looks at one of three non-player characters (NPC) and performs one of the 7 prototypic expressions; the NPC responds based on its personality. To increase the speed and efficiency of the system, the eye gaze and expression recognition modules leverage CUDA and GLSL pixel shaders.