Visible–Infrared Dual-Sensor Fusion for Single-Object Tracking

Visual object tracking is a classical yet rapidly evolving task in computer vision. Traditional RGB trackers in the visible spectrum suffer from poor lighting conditions and bad weather, which significantly affect image quality. To tackle this drawback, RGB trackers are extended from single modality (RGB) to dual modalities [RGB-thermal (RGB-T)] via visible–infrared dual-sensor fusion. Compared with visible cameras, thermal infrared cameras are insensitive to lightning conditions and have a strong ability to penetrate haze and smog. Inspired by the complementary merits of visible and thermal information, we propose a visible–infrared (RGB-T) tracker that fuses the visible modality and infrared modality on the pixel level, feature level, and decision level. First, for the pixel-level fusion, a fused histogram is derived from a four-channel RGB-T image that concatenates an aligned RGB and thermal image pair. With the fused histogram, a foreground mask is derived for guiding correlation filter learning by segmenting the foreground target from the neighbor background. Second, for the feature-level fusion, handcrafted features, such as histogram of gradient (HOG) and color name (CN), are extracted from the RGB and thermal images, respectively, and then concatenated along the channel dimension for feature fusion. Third, for the decision-level fusion, the channel weight of each feature channel in both modalities is estimated for weighting per-channel filter response in localization. To demonstrate the effectiveness of our visible and infrared fusion strategy, we perform an ablation study to evaluate each level of fusion independently on the RGBT234 dataset. We also perform extensive experiments on the RGBT234 and VTUAV_ST datasets to evaluate the performance of the proposed RGB-T tracker compared with state-of-the-art trackers. It is worth noting that our three-level visible–infrared fusion framework is generic and flexible and our implementation is far from optimal. This framework is open-source and easy to be combined with any correlation filter-based RGB-T tracker for further performance gain.