Recognition of gestures using morphological features of networks made of gesture motion images and word sequences

We propose a method to recognize human gestures using both motion images and text. The method uses two kinds of network models obtained from two kinds of sequences in a style of self-organization. We can extract both so-called common and singular parts of a gesture by analyzing the topology of the network of gesture motion images. If the order of movements in a gesture motion image matches with that of words included in a corresponding sentence, then we can also extract both so-called common and singular parts of the network model of test. The proposed method for recognizing gestures uses morphological features between two networks made of gesture motion images and word sequences. We show the usefulness of the method through an experiment using a database composed of pairs of gesture motion images and text.