Towards Cognitive State Detection using Facial Expression and Hand-over-Face Gesture

Facial expression recognition has gained attention from researchers in computer science, psychology, medicine and related fields and has grown spontaneously in recent years. However, most research focuses on posed expressions, near frontal recordings and they ignore eye gaze, head pose and considers hand occlusions as noise. Especially, in E-Learning environments, people often hold their hands near their face which occludes the face and limits the accuracy of facial expression recognition. There is empirical evidence that some of the hand-over-face gestures can be used for detection of cognitive states. In this paper, we propose to use hand-over-face gesture as novel cues and integrate facial expressions with hand-over-face gestures for recognition of cognitive states like interested, bored, unsure, happy and thinking. The proposed system is robust to variations in facial expressions, hand shapes, occlusions and performs an average recognition rate of 90.51% with 15.8 fps.