TVTracker: Target-Adaptive Text-Guided Visual Fusion for Multimodal RGB-T Tracking

Current multimodal sensor trackers mainly rely on visual cues for target tracking. However, in challenging scenarios, the visual information acquired by multimodal sensors has limited descriptive ability when the target state undergoes significant changes, which may lead to unsatisfactory tracker performance. In this work, we propose TVTracker, a two-stage tracking framework. It leverages semantic information of the target state to enhance visual cues, enabling effective RGB-T tracking. The first stage is the generation of target text descriptions. Utilizing the bootstrapping language-image pretraining (BLIP) model, we generate target textual descriptions that match the images in the dataset. In the second stage, text-guided visual fusion is performed for target tracking. Target textual and visual features are extracted separately using the text and visual branches. Then, the target textual features are integrated with the visual features to localize the target position and predict the target bounding box. In the text branch, we design the target text adaptation enhancement (TTAE) module to mitigate the interference of low-quality target textual features on visual features. In the visual branch, we develop the multimodal visual information prompters, which include the multimodal visual shared information prompter (MVSIP) and the multimodal visual shared and complementary information prompter (MVSCIP), to facilitate learning of multimodal shared or complementary visual prompts. Experiments on the LasHeR, RGBT234, and VTUAV datasets demonstrate the effectiveness of TVTracker.