SGPT: The Secondary Path Guides the Primary Path in Transformers for HOI Detection

HOI detection is essential for human-computer interaction, especially in behavior detection and robot manipulation. Existing mainstream transformer methods of HOI detection are focused on single-stream detection only, e.g.,
$image \rightarrow HOI(\mathcal{P}_{1})$
, or
$image \rightarrow HO\rightarrow I(\mathcal{P}_{2})$
. Both paths have their own characteristics of concern, so we propose a novel method, using the Secondary path
$(\mathcal{P}_{2})$
Guides the Primary path
$(\mathcal{P}_{1})$
in Transformers (SGPT). SGPT contains two core modules: the Dual-Path Consistency (DPC) module and the Instance Interaction Attention (IIA) module. DPC keeps human, object and interaction consistent on the dual-path and lets
$\mathcal{P}_{2}$
guide
$\mathcal{P}_{1}$
to learn more meaningful features. IIA fuses human and object to enhance interaction in
$\mathcal{P}_{2}$
, which allows instance to constrain interaction. Our proposed dual-path are employed during training, and only the
$\mathcal{P}_{1}$
path is used for inference. Hence, SGPT improves generalization without increasing model capacity in HICO-DET and V-COCO datasets compared to the state-of-the-arts. The code of this work is available at https://github.com/visualVk/sgpt.git.