Zero-Shot Face Recognition via Fine-Tuning CLIP with Contrastive Alignment

Face recognition has become a fundamental component in various security, authentication, and surveillance applications. However, traditional face recognition systems require extensive labeled data, suffer from limited generalization to unseen identities, and need costly retraining for new classes, thereby limiting their real-world applicability. In this study, we propose a fine-tuning approach for CLIP (Contrastive Language-Image Pretraining) models to perform face recognition using image-text pair matching instead of conventional image classification. By leveraging CLIP’s shared image-text embedding space, our method preserves its zero-shot learning capability, enabling the classification of unseen faces without additional training. This eliminates the need for large labeled datasets and frequent retraining, making it a more scalable solution. To validate our approach, we conducted extensive experiments on the face recognition task by fine-tuning the final few layers of CLIP while maintaining its zero-shot ability. Our fine-tuned model achieved a test accuracy of 77.4% when classifying unseen identities, significantly outperforming the traditional method in terms of generalization. Our approach offers a scalable, data-efficient alternative to conventional face recognition, enabling classification of unseen identities without retraining and demonstrating its suitability for dynamic, real-world applications.