Recognition of signed expressions observed by Kinect Sensor

In this paper we present an approach to recognition of signed expressions based on visual sequences obtained with Kinect sensor. Two variants of time series representing the expressions are considered: the first based on skeletal images of the body, and the second describing shape and position of hands extracted as skin coloured regions. Time series characterising isolated Polish sign language words are examined using three clustering algorithms and popular clustering quality indices which reveal natural gesture data division and indicate gesture samples difficult in further recognition. Ten-fold cross-validation recognition tests for the k-nearest neighbour classifier with dynamic time warping technique are shown. Recognition rate obtained with the skeletal image based features were improved from 89% to 95% by changing gesture representation from time series to a vector containing pairwise distances between gesture samples. The approach with skin colour based features involving utilisation of depth information of each pixel obtained by Kinect yielded 98% recognition rate.