A Novel Weakly-Supervised Proposal Method Based on Vector Perceptron and Unbalanced Action-Background Self-supervised Boundary Inferring

In weakly-supervised temporal action detection, only the category labels of action instances could be used for training. Without any temporal boundary annotations in an untrimmed video, it is a huge challenge to propose the starter and the end of an action (named as "the proposa"). In this paper, we try to focus on the temporal differences of visual contents which may change significantly when motion occurs and carry the information of temporal boundaries. First, a motion-propagating feature vector (MPV) is designed to describe different kinds of temporal differences in a video. Second, we propose a temporal difference perceptron (TDP). According to the MPV, TDP computes an attention to each video snippet to infer action areas. However, the TDP only concentrates on the most discriminate action areas and the attentions fail in slow actions. To tackle these, an unbalanced action-background self-supervised loss (USSL) function is custom to tune the distribution of activated change areas. With the unbalanced constraint, all forward changes and some backward change action regions are activated. And the self-supervised attention labels force the model to widen the gap between background and action in feature space, contributing to boundaries inferring. Our method achieves comparable performance with the state-of-arts, which outperforms STPN, an algorithm without temporal difference message, more than 2% mAP on THUMOS14, and is 20.8% (mAP@tIou=0.5) higher than UntrimmedNets on ActivityNet v1.2 which used temporal difference information roughly.