Learning to Remember Past to Predict Future for Visual Tracking

Fast and reliable adaptability to appearance variations of any target object has been the holy grail of visual tracking. Recently, Siamese-based trackers have demonstrated outstanding speed, however at the cost of adaptability and accuracy. We propose to model a temporal evolution of appearance features, allowing for adaptability without online training. Specifically, we introduce a memory-augmented convolutional recurrent neural network (RNN), named Past-to-Future (P2FNet), that takes appearance features as an input at each frame and predicts the next-frame features. RNN allows for fast adaptability to dynamically varying appearance, while the memory provides the generalization capability over longer sequences via template management. For reliability, we propose a new augmentation to train RNN to disregard corrupted features. A novel visualization method illustrates the reliable template management of the memory. The experimental results on benchmarks demonstrate the tracker shows competitive performance among real-time state-of-the-art trackers.