Real Time Hand Gesture Recognition Applied for Flight Simulator Controls

Dynamic hand gesture recognition is a desired human-computer interactions for flight simulator controls, which needs powerful enough to satisfy the requirements of high classification accuracy, fast response time. In this paper we presents a hand gesture recognition system that based on RGB camera to identify motion, hand and gesture in a sequential way. Frame difference is designed as a motion detection modality and a lightweight hand detection neural network is utilized to activate the gesture classifier. To train the system to recognize designed gestures, large gesture JESTER dataset is employed to train and test the proposed deep learning neural networks, which includes 2D ConvNets, 3D ConvNets and a fully connected layer. Experimental results show that the offline gesture classification accuracy on the JESTER dataset is 95.96% and online recognition algorithm runs on average 164 fps in the presence of hand. According to the questionnaire results after the subjects used our flight simulator system, most subjects expressed satisfaction with our gesture recognition system.