A Local Spatial–Temporal Synchronous Network to Dynamic Gesture Recognition

Dynamic skeletal data contain advanced information represented by complex compound actions and have been extensively studied in human motion recognition. Previous works usually depend on designing the traversal rules made by hand or using two separate components to obtain spatial and temporal information based on skeleton graph, respectively, which still have generalization difficulties and significant challenges in learning effective spatial–temporal information. This article proposes a local spatial–temporal synchronous network (LSTSN) for skeleton-based dynamic gestures recognition, which can simultaneously deal with spatial and temporal information in gestures. Specifically, to fully obtain the spatial–temporal information of the skeleton data, a coupled position embedding and a three-neighbor local spatial–temporal graph (LSTG) are constructed. An adaptive threshold method is designed to describe the joint connections of LSTG, and then, an attention mechanism is used to efficiently capture complex local spatial–temporal correlations. Meanwhile, multiple spatial–temporal attention modules of different periods are designed to effectively capture the LSTG’s heterogeneity. Besides, from the data aspect, a hand motion vector (HMV) is also introduced into the model to express subtle finger motions. Extensive experiment results on DHG-14/28 and SHREC’17 Track datasets demonstrate that the proposed LSTSN is competitive with state-of-the-art methods.