BR-MoE: Blind Multi-Modal Tracking with Route-Dynamic Mixture of Experts

Multi-modal object tracking combines visible light with auxiliary modalities for better tracking accuracy, including sub-tasks like RGB-T (RGB and Thermal), RGB-D (RGB and Depth), and RGB-E (RGB and Event) tracking. However, existing algorithms, designed and trained for specific tasks, impose high costs on universal tracking systems as they rely on multiple independent trackers. Despite efforts to develop universal multi-modal trackers, these methods often fail to effectively integrate both modality-agnostic and modality-specific information across various modalities, constraining their performance. In this paper, we propose the Blind Multi-Modal Tracking with Route-Dynamic Mixture of Experts (BR-MoE). This approach can simultaneously model both modality-agnostic and modality-specific features of each modality within a unified structure and model parameters, all without prior knowledge of the modality types. In particular, BR-MoE employs Vision Transformer (ViT) to extract modality-agnostic features, while embedding four parallel modality-specific experts in each ViT block to capture specific features. Additionally, BR-MoE incorporates a Modality-Aware Module (MAM) that adaptively assigns weights to the modality-specific experts based on the input and combines their outputs to generate the final feature representation. To achieve this, BR-MoE is trained in two phases. In the first, we manually provide modality types to supervise MAM and select the right expert. In the second, we do a weighted aggregation of experts’ features based on MAM’s decision and fine-tune the model for feature changes. Experiments on five multi-modal tracking datasets show BR-MoE gets state-of-the-art performance on RGBT234, DepthTrack and VisEvent datasets, and comparable performance on LasHeR and VOT-RGBD22 datasets.