HDTSLR: A Framework Based on Hierarchical Dynamic Positional Encoding for Sign Language Recognition

Sign language is the basic way for people with hearing impairment to communicate, and sign language recognition (SLR) could effectively help in this regard. Mainstream Transformer-based SLR requires positional encoding to sense the positional information of the data. However, existing PE methods globally encode the sign data result in weaken or even ignoring the sequence variation within the gestures. This article proposes HDTSLR: A Transformer-based SLR framework built on hierarchical dynamic positional encoding (HDPE) enhances individual gesture sequence features while preserving the sign overall temporal features. HDPE designs semantic positional encoding utilizing predefined scale functions with trainable biases to emphasize sign semantic relationships. The t-distribution is used by the designed lexical positional encoding to explore the unique variation of gestures. Before the HDPE operation, the sign language data is split into equal-length feature clips while feature extraction and chunking are performed by the autoencoder. The feature clips with significant changes in gesture chunk are further selected and aggregated with the remaining ones by deforming Gram matrix. In addition, HDTSLR is evaluated on the one-handed and two-handed datasets, achieving word error rates of 16.59% and 21.67%, respectively. Comparison experiments show that it outperforms known SLR methods in both accuracy and robustness.