MMFFNet: Multimodal Feature Fusion Network for RGB-T Crowd Counting

In recent years, crowd counting has received extensive attention due to its wide-ranging applications across various fields. However, the exclusive utilization of red–green–blue (RGB) images for crowd counting is susceptible to adverse factors such as lighting conditions, which significantly impair the accuracy. An increasing number of methods have emerged for crowd counting using both RGB and thermal (RGB-T) images. Nevertheless, the underutilization of the complementarity among multimodal information has hindered the progress of RGB-T crowd counting. In order to fully leverage the complementarity among multimodal information without consuming excessive computing resources, we propose a multimodal feature fusion network (MMFFNet) based on the encoder–fusion–decoder structure for RGB-T crowd counting. We put forward a feature fusion module (FFM) that employs an attention-like approach to effectively fuse the extracted features at multiple levels across RGB-T modalities. Furthermore, a feature decoding module (FDM) that primarily uses three attention mechanisms is meticulously designed, incorporating various attention mechanisms to decode the features. Extensive experiments on the RGBT-CC dataset and the DroneRGBT dataset demonstrate that our proposed MMFFNet has achieved state-of-the-art (SOTA) performance in RGB-T crowd counting. The source code and pretrained models will be released upon acceptance at https://github.com/67zhanan/MMFFNet