Detecting Adversarial Examples Using Cross-Modal Semantic Embeddings From Images and Text

Deep learning models are highly susceptible to adversarial attacks, where subtle perturbations in the input images lead to misclassifications. Adversarial examples typically distort specific features that are critical for accurate image classification. Therefore, distorting the visual semantic characteristics via adversarial attacks is challenging. Building on this insight, we propose a method that leverages a multimodal space for detection of adversarial examples. Our approach utilizes cross-modal semantic embeddings derived from the images and labels. We employed image and text encoders trained to maximize the similarity between the embeddings extracted from images and labels, optimized these cross-modal semantic embeddings and mapped them to a common semantic space to extract detailed semantic features. When the embeddings extracted from an image and its label are semantically similar, they are closely positioned in this space. Based on these observations, we compared the similarities between the embeddings of the input images with those of their corresponding labels to detect adversarial examples. We evaluated our proposed method on classification benchmark datasets and demonstrated detection rates of over 91.75% for various adversarial attacks. Moreover, we determined that the ground truth had higher similarity than the other label to the detected adversarial sample, enabling robust predictions with the most similar label.