Co-Inference Discriminative Tracking Through Multi-Task Siamese Network

In essence, visual tracking is a matching problem without any prior information about a class-agnostic object. By leveraging large scale off-line training data, recent trackers based on Siamese networks usually expect to pre-learn underlying similarity functions before a tracking task even begins. Consequently, they lack discriminative and adaptive powers. To address the issues, we propose a multi-stage co-inference tracker (named MSCI) via a multi-task Siamese network, in which a complicated tracking task is divided into three complementary sub-tasks (i.e., classification, regression and detection). Firstly, we design a novel multi-task loss function to end-to-end train the multi-task Siamese network via jointly learning from three sub-tasks. The multi-task Siamese network contains three parallel yet collaborative output layers, which correspond to three key components of our tracker (i.e., classifier, regressor and residual learning based detector). By sharing representations within the components, we not only improve each component’s generalization performance, but also enhance our tracker’s discriminative power. Then, we design a co-inference approach to effectively fuse the complementary components. As a result, our tracker can avoid the pitfalls of purely single components and get reliably observations to improve its adaptive power. Comprehensive experiments on OTB2013, OTB2015 and VOT2016 validate the effectiveness and robustness of our MSCI tracker.