HySSTP: Hyperspectral Video Tracker Embedding Multi-Modal Spatial-Spectral-Temporal Features

Hyperspectral video (HSV) delivers extensive spatial, spectral, and temporal cues essential for sensing intricate object variations and addressing the shortcomings of conventional multi-modal and RGB video tracking, such as multi-device imaging and modality alignment. However, current HSV tracking methods often face challenges, including data anxiety, band gaps, large data volumes, and weak temporal cues, leading to suboptimal performance. To this end, we introduce the hyperspectral video tracker embedding multi-modal spatial-spectral-temporal features (HySSTP). We first develop a modal generation adapter to leverage the explainability of integrating machine and physical patterns, effectively bridging band gaps. Next, we create a spatial-spectral encoding adapter for adaptive interaction among multiple modalities. Finally, a temporal decoding adapter is designed to enhance the spatial, spectral, and temporal features, capturing both dynamic and static object characteristics. HySSTP introduces fewer trainable parameters and demonstrates superior performance in extensive evaluations. The code will be available at https://github.com/YZCU/HySSTP.