Homomorphic Encryption-Based Facial Recognition for Dual-Layer Similarity Analysis of Modified Images

Recently, advanced closed-circuit televisions have been widely installed for security and crime prevention purposes, leading to the large-scale adoption of facial recognition systems. However, the indiscriminate use of such technologies has resulted in excessive intrusion into individuals’ private lives, prompting ongoing discussions around privacy infringement. In response, Homomorphic Encryption (HE)-based facial recognition technologies—which aim to ensure both privacy and security—have been actively researched and developed. Nevertheless, existing researches have limitations in that they did not seriously consider the possibility of original image leakage and failed to incorporate color information into the similarity measurement process. To address these issues, we propose a mechanism that considers not only facial landmarks but also color information in facial recognition, applying HE in a way that causes actual image modification in the facial area. This ensures that privacy can still be protected even in the event of facial data leakage. In the proposed mechanism, similarity values are independently derived from both facial landmarks and aligned color information. After decrypting each homomorphically encrypted value, a final Dual-Layer similarity is computed by combining both, enabling the identification of the facial image with the highest similarity. To achieve highly accurate and reliable final similarity computation, we propose a Dual-Layer similarity derivation formula based on the Softmax function, which is widely used as a core element in deep learning for multi-class classification. The effectiveness and necessity of the proposed method are demonstrated through comparisons with the arithmetic mean, weighted arithmetic mean, and harmonic mean. Furthermore, since most existing HE schemes are limited to supporting addition and multiplication operations, we analyze similarity derivation algorithms that comply with these constraints. Through a comparative analysis based on computational efficiency, we demonstrate that Euclidean Distance is well-suited for efficient HE-based facial recognition.