Deepfake Detection Using Spatio-Temporal-Structural Anomaly Learning and Fuzzy System-Based Decision Fusion

Deepfake techniques have been evolving rapidly in recent days and pose a severe security threat. Detecting such generated fake videos is a challenging task. Existing deep learning based deep fake detection model struggle in identifying fake videos when experimented on challenging dataset. In this paper, we propose a Fuzzy logic based Deepfake Detector system (Fuzzy-DFD) using a deep learning model ResNet-50 as encoder. The system workflow includes preprocessing, frame extraction, deep fake detection and decision fusion. In preprocessing, frames are extracted from video, facial region is cropped, and frame resizing is performed. Next, optical flow maps are generated using SEA-RAFT model and edge defined frames generated using Frei-Chen mask technique. These three frame variants present a rich representation of video for the deep learning model. 3 ResNet-50 models are employed as encoders to generate feature maps trained using above frame types, ensuring anomaly identification in spatial, temporal and structural domains. A novel three input and one output Fuzzy Inference System is designed for decision fusion combining the detection results of all three classifier models to make decision on video being a fake. The performance of the proposed Fuzzy-DFD system is evaluated by training the model on well-known deep fake classification datasets namely FakeForensics++ and Celeb-DF(v2). Further results are compiled and evaluated using performance indicators like accuracy, Precision, F1-Score, Recall and AUC. Fuzzy-DFD system attained accuracy of 99% and 93% for FaceForensics++ and Celeb-DF(v2) datasets respectively. Comparison study with other SOTA deep fake detection models is conducted, and Fuzzy-DFD significantly outperformed the other models taken for comparison. The anomaly learning through RGB, optical flow and structural characteristics with fuzzy system-based decision fusion technique contributes to a robust deep fake detection system.