Adaptive Hierarchical Feature Difference Auto-Encoder for Robust RGB-T Object Tracking

RGB-T object tracking leverages visible and thermal infrared modalities to enhance robustness in challenging environments. While deep learning-based RGB-T trackers predominantly use feature-level fusion, pixel-level fusion remains underexplored. This paper introduces the Hierarchical Feature Difference Auto-Encoder (HFDAE), a novel pixel-level fusion approach that refines the RGB modality before tracker input. HFDAE adaptively enhances RGB content using hierarchical TIR features, dynamically emphasizing object saliency. HFDAE consists of three key components: (1) a shallow RGB autoencoder that preserves structural and color information, (2) a TIR encoder with variable-depth decoders generating hierarchical TIR representations, and (3) a fusion module that integrates salient thermal features into the RGB image. Salient features are extracted by computing differences between hierarchical and base TIR images, which are then added to the base RGB image to generate the final fused output. Unlike conventional pixel-level fusion methods, HFDAE is optimized directly for tracking, learning fusion strategies without predefined modality assumptions. Extensive experiments on benchmark datasets demonstrate HFDAE’s superior tracking accuracy and robustness across diverse scenarios. The proposed approach improves the base tracker’s precision rate by approximately 10%. Code is available at https://github.com/mohamed-e-awad/HFDAE.