Fragile fingerprint for protecting the integrity of the Vision Transformer

Nowadays, with the rapidly development of deep learning, deep learning models have been widely deployed in various fields and generated significant commercial interest. Some technology companies upload their trained models to cloud servers and serve them to the end-users. Many works have shown that the convolutional neural networks are vulnerable to some model modification attacks, which raise concerns about integrity authentication of the convolutional neural models. Additionally, Transformers based on attention mechanism are now commonly used in computer vision applications, and the need to verify the integrity of ViTs arises if the ViT model is deployed in the safety critical systems. In this paper, we propose a fragile fingerprint method for verifying the integrity of the ViTs, which is based on the targeted adversarial examples. Compared with the existing works, the proposed fingerprint method does not modify the ViTs. We generate some fragile fingerprints, which are classified as the targeted label. In the verification stage, if the fingerprints are successfully classified as targeted label with 100% success rate, we can claim that the ViTs is not modified. Otherwise, when the fingerprint verification success rate is lower than 100%, we can claim that the integrity of ViTs is compromised. Experimental results demonstrate that the fingerprints can effectively verify the integrity of the ViTs when the ViTs is modified by model attacks, even though only a small number of weights of ViTs are changed.