HARG: Hierarchical Adaptive Reasoning Graph for Activity Parsing

As a video understanding task, activity parsing aims at encompassing actions into multiple levels of activity components, including activity, sub-activity and atomic action, enabling understanding of complex video scenes within multimedia systems. Existing methods form activity parsing as a multi-task learning problem to predict multi-granular activity labels simultaneously, which ignores modeling the hierarchical structure and the fine-grained transitions of activity components at different levels. In this paper, we propose a Hierarchical Adaptive Reasoning Graph (HARG) to model the hierarchical structure (i.e., object level $\rightarrow$ atomic action level $\rightarrow$ activity level) dynamically and precisely. To achieve that, an object reasoning graph (ORG) and an atomic action reasoning graph (ARG) are designed to reason fine-grained information transitions between multiple actors at different levels. In addition, an adaptive segmentation module (ASM) is investigated for bridging the gap among different levels, permitting step-by-step reasoning from the object level to the atomic action level. Experimental results show our method outperforms state-of-the-art methods on two activity parsing datasets, achieving hierarchical modeling and fine-grained reasoning for activity understanding. The code is available on GitHub: https://github.com/whuoyj/HARG.