RGSA: Sensor-Based Group Activity Recognition Model With Relation Gating and Spatiotemporal Attention

Sensor-based group activity recognition (GAR) is a challenging task that requires handling complex individual actions and interindividual relations. An imperative exists to explore a relation learning framework for effectively modeling and dynamically extracting group-relevant actions and interactions in group activities. To solve this issue, we present relation gating and spatiotemporal attention (RGSA), a novel model for sensor-based GAR with RGSA. RGSA employs a graph structure to model individual interactions achieving dynamic updating of the group relation graph through the graph-passing process. By introducing a relation gating mechanism and proposing the interaction feature correlation reward, RGSA selects group-relevant relations through deep reinforcement learning. Subsequently, an enhanced set of relations is obtained. In addition, RGSA introduces a temporal attention mechanism. This mechanism assesses the coherence of actions by calculating the consistency between an individual action at a specific timestep and the entire group activity. It quantifies the contribution of the action at that timestep, helping to exclude the interference of irrelevant actions. Meanwhile, RGSA introduces a spatial attention mechanism that selectively integrates and stores interactional information between neighboring individuals based on the contribution of their interactions. It facilitates a more comprehensive capture of the dynamic features of group activity. Experiments are conducted on two datasets, revealing that RGSA proficiently identifies group activities while effectively suppressing interference from noncorrelated relations and irrelevant actions in GAR. It ensures a sustained high accuracy in recognizing group activities even in the presence of interference.