Heterogeneous Face Recognition Based on Multiple Deep Networks With Scatter Loss and Diversity Combination

Due to the gap between sensing patterns of different domains and a lack of sufficient training sample, heterogeneous face recognition (HFR) is still a challenging issue in the computer vision community. In this paper, we propose a novel method called multiple deep networks with scatter loss and diversity combination (MDNDC) for solving the HFR problem. As we know, the performance of deep models is affected by data, network structure, and loss function, so we devote much effort to improve the HFR performance from all these three aspects. First, to reduce the intra-class variations and increase the inter-class variations, the scatter loss (SL) is used as an objective function that can bridge the modality gap while preserving the identity information. Second, we design a multiple deep networks (MDN) structure for feature extraction and propose a joint decision strategy called diversity combination (DC) to adaptively adjust the weights of each deep network and make a joint classification decision. Finally, instead of using only one publicly available dataset, we make full use of multiple datasets to train the networks, which can further improve the HFR performance. The extensive experiments are carried out on two challenging NIR-VIS HFR datasets, CASIA NIR-VIS 2.0 and Oulu-CASIA NIR-VIS, demonstrating the superiority of the proposed method.