Tracking objects in 3D space with back-projected correlation filter

RGB-D cameras which become popular in recent years have brought new opportunities as well as challenges to many computer vision tasks. However how to fully make use of the depth information in visual tracking is still under-investigated in current literatures of visual tracking. We present in this paper an effective method for object tracking in RGB-D videos. In our approach, depth information is integrated into the kernelized correlation filter (KCF) tracking framework. Instead of treating depth as a color channel or adopting a volumetric representation of target, we take advantage of the 3D information of the scene behind depth and back-project the 2D response map of conventional KCF into 3D space. Then the 3D location of target can be determined by mode seeking in the resulted 3D response map. Since 3D location is obtained, the scale of the template can be determined naturally which in turn can benefit the KCF. We also propose a model updating mechanism which can balance model adaption against tracker drifting. Experimental results have demonstrated the effectiveness of the proposed method.