Video Visual Relation Detection Based on Trajectory Fusion

Nowadays, many video visual relation detection models rely on object tracking. However, detecting a target’s long trajectory in a raw video is still an open research issue, as tracklet-based methods suffer from poor performance in detecting the long-term relationship in the video. In order to solve this problem, we design a multi-trajectory fusion strategy that fuses the trajectory and relieves the track alignment strain. Then, to enhance encoder and decoder learning, we design a multi-branch encoder layer that separates the matching information from the appearance information of the trajectory and exploits the matching information to promote the attention maps in the decoder layer. Extensive experiments on two benchmarks, ImageNet-VidVRD and VidOR, demonstrate our model’s effectiveness. The developed model(TFPDT) improves VidSGG’s performance over the existing trajectory quality (about 1.64 percent mAP in Relation Detection on VidVRD and 2.7 percent mAP on VidOR).