HyperTrack: A Unified Network for Hyperspectral Video Object Tracking

Compared to conventional RGB images, hyperspectral images offer a more comprehensive range of spectral information, encompassing both visible and infrared bands. This enhanced spectral information facilitates trackers in effectively differentiating the target object from background clutter, realizing more robust recognition. However, hyperspectral video cameras exhibit variability in terms of spectral ranges and band numbers, producing distinct modalities. Developing specialized networks for each modality proves to be inefficient and time-consuming, while networks trained for one modality struggle to perform well on others, generating unsatisfactory outcomes. To confront these challenges encountered in various tracking scenarios, HyperTrack is proposed as a unified object tracking network tailored for hyperspectral videos. The proposed HyperTrack can be employed individually for single object tracking across three different modalities of hyperspectral videos: near infrared (NIR), visible (VIS), and red-to-near infrared (RedNIR). Specifically, since hyperspectral data have multiple bands, a band gate module is introduced into the network to enable it to select bands from hyperspectral images, thereby reducing the dimensionality for hyperspectral images. Furthermore, in order to effectively utilize the variability amongst the three different modal data, a computationally simple band embedding module is introduced to improve the object tracking performance of different modal hyperspectral videos. Additionally, a hybrid attention module is devised to efficiently extract and interact features between the template and search at each stage. As a unified network, HyperTrack achieves state-of-the-art comprehensive results across three different types of hyperspectral videos, particularly excelling with VIS and NIR type data. The code and models are publicly available at https://github.com/supertyd/HyperTrack.