NOT-156: Night Object Tracking using Low-light and Thermal Infrared: From Multi-modal Common-aperture Camera to Benchmark Datasets

Night object tracking (NOT) is aimed at tracking objects under low-illumination conditions at night. Existing works concentrate on thermal infrared modality, while some RGB-T data also contains night scenes. However, night scenes in these datasets are mostly well-lit, making it challenging to fully cover low-illumination scenarios. In this paper, we focus on the night object tracking task, and build up a novel low-light visible and thermal infrared (LOL-T) multi-modal benchmark dataset for Night Object Tracking (NOT-156). To achieve night vision, we design a common-aperture LOL-T camera by integrating a highly dynamic low-light visible imaging sensor with a thermal infrared sensor in a common aperture optical system. The proposed dataset consists of 156 video sequences and a total of 170k annotated frames, including various low-illumination night scenes such as dark room, street, corridor and so on. Compared with existing datasets, the NOT-156 has more comprehensive and distinctive attributes (thermal variation, noise, high illumination overexposure, etc.). Comprehensive experiments are carried out to evaluate the performance of the advanced visible, infrared, and visible-thermal trackers on the proposed NOT-156 dataset. The authors believe that NOT-156 has great potential in application and development of night vision. The dataset will be made available at: http://rsidea.whu.edu.cn/NOT156_dataset.htm.