Hierarchical Cross-Modal Image Generation for Multimodal Biometric Recognition With Missing Modality

Multimodal biometric recognition has shown great potential in identity authentication tasks and has attracted increasing interest recently. Currently, most existing multimodal biometric recognition algorithms require test samples with complete multimodal data. However, it often encounters the problem of missing modality data and thus suffers severe performance degradation in practical scenarios. To this end, we proposed a hierarchical cross-modal image generation for palmprint and palmvein based multimodal biometric recognition with missing modality. First, a hierarchical cross-modal image generation model is designed to achieve the pixel alignment of different modalities and reconstruct the image information of missing modality. Specifically, a cross-modal texture transfer network is utilized to implement the texture style transformation between different modalities, and then a cross-modal structure generation network is proposed to establish the correlation mapping of structural information between different modalities. Second, multimodal dynamic sparse feature fusion model is presented to obtain more discriminative and reliable representations, which can also enhance the robustness of our proposed model to dynamic changes in image quality of different modalities. The proposed model is evaluated on three multimodal biometric benchmark datasets, and experimental results demonstrate that our proposed model outperforms recent mainstream incomplete multimodal learning models.