TAPTR3D: Decoupled 3D Point Tracking Boosts 2D and Further Enhances 3D Tracking Accuracy

In this paper, we propose TAPTR3D to extend Track Any Point Transformer (TAPTR) to 3D. Existing methods extend 2D point tracking models to track in 3D by simply appending a depth prior to the target tracking point's feature. However, such a design brings a misalignment between the target point and video frames in the feature space, degrading the fundamental 2D localization ability, and ultimately undermining 3D point tracking. To address this issue, we propose to decouple the localization of a point in 3D space into two cascaded sub-tasks. First, localize the target point in the 2D image plane, yielding not only a well-refined 2D point descriptor but also a more reliable depth prior sampled on the depth map according to the refined 2D position. Based on these more accurate 2D and depth information, the second sub-task, localizing along the depth axis, becomes simpler. Benefiting from this decoupled formulation, training on the 3D point tracking task not only avoids degradation, but also improves 2D point tracking performance, which further leads to better 3D tracking results. To further improve the accuracy during localizing on 2D image plane, we propose Depth-enhanced Cross Attention, which jointly considers 2D semantic features and the 3D spatial priors from the depth map to enhance the quality of the attention weights during cross attention, leading to better interaction with video frames. Meanwhile, to further enhance the robustness during localizing along the depth axis, we propose Local Geometry-aware Depth Track, which samples on the depth map in a local grid form to construct depth priors enriched with 3D geometric information, making the depth tracking more resilient to the inherent noise in the estimated depth maps. TAPTR3D achieves the best performance on most datasets, validating its effectiveness.