Real-Time Infrared Small Target Detection With Nonlocal Spatial-Temporal Feature Fusion

Infrared small target detection is a challenging task in which many researchers have made lots of achievements. While the performance of single-frame (SF) detection is still limited due to the lack of usage of multiframe (MF) continuous information, many spatial-temporal detection methods have been developed. However, most algorithms need to register the image or feed a set of images as the input. Inputting a batch of group images usually leads to large computations, which heavily affects their real-time processing capability in resource-limited machines. To tackle the problem, we propose a nonlocal multiframe network (NLMF-Net) with only a few additional computations (no more than 0.01 GFLOPs) compared to the SF baseline while achieving significant performance improvements. The NLMF-Net correlates features from grid cells with high confidence between current and past frames. While most background grid cells are removed after the SF processing, the MF feature fusion only focuses on a few potential target grid cells, resulting in high computation efficiency. The proposed vector length similarity module enlarges the difference between different grid cells and the non max similarity suppression further suppresses the backgrounds during the fusion, promoting the MF performance. The NLMF-Net can be readily deployed on Jetson Nano at a speed of 20 FPS for 288 × 384 image processing or Mi Pad 2 with a speed over 35 FPS for 128 × 128 part image processing. Extensive experiments show that our proposed method achieves state-of-the-art performance on three datasets while maintaining high efficiency in a real-time processing manner.