EMTrack: Event-Guide Multimodal Transformer for Challenging Single-Object Tracking

Object tracking is an important visual task in the field of remote sensing. Currently, traditional frame-based paradigms have demonstrated superior performance. However, research on remote sensing object tracking by applying event cameras has not yet been conducted. Meanwhile, there are still four key issues that need to be addressed: 1) remote sensing object tracking is prone to losing the target in challenging scenarios (e.g., occlusion, motion blur, overexposure, and low light); 2) mainstream event-to-tensor representation methods lack detailed spatiotemporal cues, making it difficult to achieve better integration with exposure frames; 3) the widely used vision transformer (ViT) backbone consumes significant computational resources and incurs increased inference time, particularly when simultaneous feature extraction for both search and template; and 4) unimodal information (event-based or frame-based) struggles to address all complex environments encountered in object tracking. In this work, we introduce the remote sensing event-based object tracking (RSEOT) dataset, the first multimodal remote sensing dataset based on both event and visible information, which encompasses various challenging scenarios while offering a new insight into object tracking. In addition, we have improved the voxel grid by applying linear integration to the preceding and following exposure events, facilitating better integration with frame-based paradigms. Furthermore, we propose a novel backbone, ViT-STCA, while introducing search-template cross attention (STCA) to merge the granular features of search and template. This approach significantly improves tracking performance while greatly reducing computational complexity. Extensive experiments on the RSEOT dataset demonstrate that our method achieves 29.60% area under curve (AUC), $46.02\% P$ , and $36.11\% P_{\text {norm}}$ , outperforming all other state-of-the-art single-object tracking approaches (e.g., SUTrack, SeqTrackv2, One Tracker, SDSTrack, Un-Track, ViPT, and ProTrack). The code and model are available at https://github.com/lpsg555/EMTrack