Hand Detection and Gesture Classification based on Transformer

Hand detection and gesture recognition are an important branch of artificial intelligence, which are widely used in daily life, such as interaction between human and machines, communication using signals, virtual reality and so on. In order to realize gesture recognition in unconstrained situations, we propose a novel end-to-end model based on Transformer, which can detect multiple hands in a picture. Our network can be roughly divided into the following three stages. In the first stage, we will extract feature maps of different layers by convolutional network from monocular RGB images. In the second stage, features are sent into the codec module based on Transformer, and hand detection is further carried out through multi-layer perceptrons with the decoding vectors as the input. In the third stage, we combine the attention weight maps obtained in the second step with the features of different layers in the first step to complete the gesture recognition task. From the experiment results, our network accurately identifies the positions of multiple hands in the picture and correctly recognizes hand gestures.