Detecting Black-Box Model Probing Attacks Through Probability Scores

In the black-box model probing attack, the attacker sends a series of model inference requests to a victim model to map out the classification boundary of the model. This attack is considered critical because it helps the attacker gain a better understanding of the model and launch follow-up attacks. While most existing works defend against this attack through the input of the victim model, this paper addresses the problem by utilizing the output of the victim model, i.e., the sequence of probability scores generated through the model inquiries. Our research discovered that the sequences of top-2 probability scores generated by different model probing attacks form distinct patterns that can be detected through time-series classification methods, such as Transformer and Bidirectional Long Short-Term Memory (BiLSTM). Our experiments showed that both Transformer and BiLSTM can detect known and unknown black-box model probing attacks and their variants. Compared with existing defense methods, our classifiers can reduce the Attack Success Rate (ASR) of the OARS-enhanced attacks from 93% to 1.78%. To facilitate further studies on the black-box model probing attack, we collected the probability score sequences generated by 6 different black-box model probing attacks and released them as an open dataset: https://www.kaggle.com/datasets/drvoyager/model-probing-attack-dataset.