Machine Learning Aided Minimal Sensor based Hand Gesture Character Recognition

Hand gesture recognition is the process of detecting the hand movements via sensor measurements for detecting an activity, such as writing a letter or a number. Recognising the handwritten characters using wearable devices enables machine-human interaction to occur without the need for a communication method. An intelligent automated framework is required to accurately detect the handwritten characters using wrist worn sensor signals, in particular, with minimal number of sensors. Moreover, the system developed needs to have the capacity to recognise the characters written in different sizes. In order to address these, we analyse performance of several machine learning models using single/multiple sensors namely, accelerometer or/and gyroscope, for recognising hand gesture characters including alphabet and numbers of varying sizes. We formulate a set of features that enable robust and accurate detection of the characters.We performed novel data collection using an off-the-shelf wrist-worn sensor based device, and evaluated our framework to detect the different characters effectively. The maximum accuracy (90.40%) was achieved using both sensors and Random Forest (RF) model. This was dropped to 82.51% for the same model using accelerometer sensor alone. Using the gyroscope sensor, an overall average accuracy of 80.16% was achieved with the Forward Neural Network (FNN) model. Although the model based on both sensors showed the best performance, our evaluation reveals that it is feasible to develop a machine learning model using single sensor to detect hand gesture characters of varying sizes with reasonable (≥ 80%) accuracy.

I. Introduction

Gesture recognition is important for developing effective human-computer interfaces [1]. Gestures are the movements of human body used for communication, such as hand movements, head nodding and eye movements [2], [3]. Among the various types of gesture recognition systems that exist in the literature, the handwritten character recognition using wearable devices has become an active research area in the recent times. Hand gesture recognition helps people with hearing disability and children with learning difficulties to communicate effectively [2], [4]. For example, developing an educational quiz that can use gesture based written character recognition for children suffering from dyscalculia (a condition that hampers the learning capability). Moreover, with the increased availability of wearable sensor devices commercially, which are small and convenient to use during the daily living conditions, it has become possible to utilise these to recognise gestures in real-time. In particular, hand movements can be measured in high resolutions and in real-time using off-the-shelf wrist-worn watch-based sensors.