Data and Algorithms for End-to-End Thermal Spectrum Face Verification

Despite recent advances in deep convolutional neural networks (DCNNs), low-light and nighttime face verification remains challenging. Although state-of-the-art visible-spectrum face verification methods are robust to small changes in illumination, low-light conditions make it difficult to extract discriminative features required for accurate authentication. In contrast, thermal face imagery, which captures body heat emissions, captures discriminative facial features that are invariant to lighting conditions, enabling low-light or nighttime recognition performance. However, due to the increased cost and difficulty of obtaining diverse thermal-spectrum data, directly training face verification systems on small thermal-spectrum datasets results in poor verification performance. This paper presents a synthesis-based algorithm that adapts thermal spectrum face images to the visible spectrum, allowing us to repurpose off-the-shelf visible-spectrum feature extractors without fine-tuning. Our proposed approach achieves state-of-the-art performance on the ARL-VTF dataset. Importantly, we study the impact of face alignment, pixel-level correspondence, identity classification with label smoothing, and synthetic data augmentation for multi-spectral face synthesis and verification. We show that our proposed method is widely applicable, robust, and highly effective on the ARL-VTF dataset. Finally, we present MILAB-VTF(B), a multi-distance, unconstrained thermal-visible dataset. To the best of our knowledge, it is the largest, most diverse dataset of its kind, collected in realistic conditions. We show that our end-to-end thermal-to-visible face verification system serves as a strong baseline for the MILAB-VTF(B) dataset.