Explainable Deepfake Detection with Human Prompts

Facial manipulation techniques pose a significant threat to society due to the prevalence of deepfake content on the internet. While previous efforts have focused on developing accurate deepfake detection models, these models may be limited in real-world scenarios due to the lack of confidence that human analysts have in their results. Therefore, this study presents a novel approach to improve the practicality of deepfake detection models by incorporating human understanding. We propose a human prompt based deepfake detection framework that overlays Human-enhanced artifacts attention onto image artifact attention, which utilizes vision prompts to improve the model’s responsiveness and feedback ability while preserving its precision and generalizability. The deepfake detection model achieves an AUC score of 0.99 on the FaceForensics++ dataset and exhibits graceful generalization when evaluated on the Celeb-DF dataset. Furthermore, the model generates "possible area of manipulation" that provides an intuitive signal to facilitate interpretation of the detection process, bridging the gap between machine and human perception of "fake". Our proposed approach can potentially mitigate the harm caused by deepfakes and provide a more reliable solution for real-world applications.