SIA-GAN: Scrambling Inversion Attack Using Generative Adversarial Network

This paper presents a scrambling inversion attack using a generative adversarial network (SIA-GAN). This method aims to evaluate the privacy protection level achieved by image scrambling method. For privacy-preserving machine learning, scrambled images are often used to protect visual information, assuming that searching the scramble parameters is highly difficult for an attacker due to the application of complex image scrambling operations. However, the security of such methods has not been thoroughly investigated. SIA-GAN learns the mapping between pairs of scrambled images and original images, then attempts to invert image scrambling. Therefore, the attacker is assumed to have real images whose domain is the same as that of scrambled images. Experimental results demonstrate that scrambled images cannot be recovered if block shuffling is applied as a scrambling operation. The experimental code of SIA-GAN is available at https://github.com/MADONOKOUKI/SIA-GAN.


I. INTRODUCTION
Deep neural networks (DNNs) have produced impressive results for various computer vision tasks [1]- [3] owing to the rapid advancement of neural networks. DNNs can satisfy user demands for training on personal data. However, personal images often contain sensitive information (e.g., faces, addresses, social relationships). Therefore, privacy protection is important to develop DNN solutions. Image scrambling [4]- [7] has been introduced to protect the privacy of visual information, and it enables privacy-preserving DNN training in external computing environments (e.g., cloud, shared server). In addition, image scrambling has low memory requirements and a low computational cost compared with other methods such as homomorphic encryption [8]- [13]. Therefore, it is suitable for privacy-preserving machine learning. Figure 1 shows the diagram of image scrambling using extended learnable encryption (ELE) [6], which simply performs pixel shuffling (i.e., block shuffling and a block-wise pixel operation). By applying pixel shuffling, representative features for The associate editor coordinating the review of this manuscript and approving it for publication was Prakasam Periasamy . classification can be extracted from scrambled images while hiding visual information [8]- [13].
The development in [6] is a basis for our study. It shows that an adaptation network notably contributes to maintain the classification performance because features should be extracted from scrambled images. Block shuffling, an image scrambling method, effectively hides visual information, and an adaptation network is required to obtain high classification performance for model inversion. Therefore, a scrambled image classification framework can provide a secure solution for machine learning in external computing environments. Nevertheless, image scrambling should be further investigated regarding security against unexpected attacks for its deployment in real settings.
Cryptanalysis methods [17]- [19] are proposed to evaluate the security level of permuted images. These works use the correlation and histogram of images for inversion. Those cryptanalysis methods are only for gray-scale images permuted pixel location. For that reason, those methods cannot be directly applicable to the scramble image approaches [4]- [7] because the scramble image approaches are for the RGB images, and they apply not only pixel location permutation but also channel-wise shuffling and negative-positive transformation. VOLUME 9, 2021 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ FIGURE 1. Diagram of ELE [6] for image scrambling. It comprises two main operations, block shuffling and a block-wise pixel operation.

FIGURE 2.
Scenario of machine learning using image scrambling under compromised cloud and data leaked to attacker. During training, the user sends pairs of images and labels to the cloud. During inference, a posterior probability is calculated from a scrambled image in the cloud. The attacker can steal a pair of images and labels during training and images during inference.
Recently, generative adversarial network (GANs) [20] have been applied to model inversion attacks [21]. Such attacks aims to estimate sensitive information in the training data of machine learning models. In addition, they use partial public information, which can be very generic, to learn the prior distribution of a real image through a GAN for guiding the inversion process, thus avoiding the reconstruction of private training data from scratch. Using blurred or corrupted private images, the original private images can be mostly recovered through a model inversion attack. However, attacking scrambled images is challenging because they completely hide visual information, outperforming blurred or corrupted images.
We propose a scrambling inversion attack using a GAN (SIA-GAN) that targets scrambled images using an adaptation network, to recover semantic information. The proposed SIA-GAN can be used to evaluate the effectiveness of image scrambling methods. SIA-GAN minimizes the divergence in feature extraction from scrambled images and real images, whose contents are similar to those of images before scrambling.
Experiments confirmed that SIA-GAN effectively attacks scrambled images, and our novel adaptation network provides a high performance for GAN-based attacks. Furthermore, SIA-GAN can attack reference images whose distribution is different from that of the target images to be revealed. The experimental results show that block shuffling is one of the most secure methods to generate robust scrambled images.
This study provides contributions in the following aspects: • Introduction of a simple and efficient method, SIA-GAN to evaluate the security of scrambled images, • Effective training by integrating an adaptation network into a GAN, • Qualitative evaluations of visual information hiding in terms of learned perceptual image patch similarity (LPIPS). The remainder of this paper is organized as follows. The scenario considered in this study for image scrambling machine learning is presented in Section II. The evaluated image scrambling methods are described in Section III-A. The proposed SIA-GAN is detailed in Section IV. The evaluation results of SIA-GAN are reported in Section V. Finally, Section VI concludes this paper. SIA-GAN allows to verify the privacy protection level of scrambled images, and block shuffling can effectively protect sensitive information in scrambled images. Figure 2 shows the scenario for model training considered in this study. During training, the user prepares a classification network and image storage in the cloud. When the users sends an image-label pair of training data, image scrambling is applied to protect conceal visual information. The classification network is, then, trained using the labeled scrambled images. During inference, the user sends a scrambled test image, and a posterior probability is inferred by the classification network in the cloud and retrieved to the user.

II. EVALUATION SCENARIO
When data are transmitted in a public computing environment, an attacker may steal data from training or testing. In this case, the attacker can obtain the following information: • Scrambled images used for training or testing • The block size of scrambled images • Class information (e.g., dog, cat) Scrambled images can be intercepted when the user sends them to the cloud. In addition, the block size can be obtained when the user constructs an adaptation network, and class information can be obtained from the labels. Then, real images can be crawled using both the class information and intercepted images through methods such as applying a GAN for image scrambling inversion.

III. RELATED WORKS A. IMAGE SCRAMBLING METHODS
Various image scrambling methods have been proposed to prevent cyberattacks to machine learning methods implemented in external computing environments [4]- [7]. Image scramble relies on diverse operations to render visual image features imperceptible to humans. Figure 1 shows an overview of ELE image scrambling [6]. First, an input image is divided into blocks. Then, the block locations are shuffled. In each block, the intensity location is shuffled. Finally, the scrambled image is obtained by integrating the blocks. Table 1 lists the characteristics of existing image scrambling methods. Pixel-based image encryption (PE) [14] uses the negative-positive transform and color component shuffling with a unique key per pixel. Random PE [15] also uses the negative-positive transform and color component shuffling with a key pixel, but the keys are generated at every execution of scrambling. Therefore, the scramble key is not restored. Learnable encryption (LE) [4] shuffles pixels and applies negative-positive transforms with the same key for every block. Then, each block is split into eight bits, four upper and four lower bits, to generate from three to six block channels. Subsequently, the pixels are randomly shuffled, yielding diverse combinations. The intensity of the randomly selected pixels is then reversed, yielding more combinations. ELE [6] uses block-wise pixel shuffling with a unique key per block and block location shuffling. Encryption  then compression (EtC) [16] uses block rotation followed by inversion, negative-positive transform, color component shuffling, and block location shuffling with the same key for every block.
The abovementioned image scrambling methods hide sensitive information in an image because their underlying operations completely distort visual information in an image. In this study, we confirmed that these methods are less secure under a GAN-based attack if the partial public information related to the scrambled images is obtained. Although restoring the original state from a scrambled image is an ill-posed problem, the proposed SIA-GAN may solve it.

B. CRYPTANALYSIS
Cryptanalysis methods [17]- [19] have been to evaluate the security of permuted images that aims at ensuring security.
These works invert the original state of permuted images using a greedy searching way. They use gray-scale images that only permute the pixel values at the spatial space. The abovementioned cryptanalysis methods use the correlation and histogram of images for inversion. In the case of scrambled images, these methods are difficult to apply since the scrambled image is an RGB image and has other shuffling operations such as channel-wise shuffling and negative-positive transformation. Considering this point, our proposed attack aims at inverting the original state from such a scramble image. Since we use a data-driven approach, the proposed SIA-GAN can naturally learn the original state from the image distribution. In addition, we can quickly invert the original state of scrambled images since the proposed SIA-GAN only needs a forwarding time.

IV. PROPOSED SIA-GAN
This section introduces the proposed SIA-GAN, which aims to unveil vulnerabilities in image scrambling methods. To successfully attack a scrambled image using a GAN, we use an adaptation network in the generator to learn the image scrambling operations SIA-GAN aims at demonstrating the dangers of scrambled images. To successfully attack the scrambled images, we use an adaptation network in generator that can learn the way of image scrambling.

A. OVERVIEW
SIA-GAN is proposed to attack against the scrambled images by restoring the original information through model inversion. Figure 3 shows an overview of SIA-GAN, which consists of a generator and a discriminator. The generator comprises an adaptation network and a feature decoder. The details of the adaptation network, feature decoder, and discriminator are listed in Tables 4, 5, and 6, respectively. SIA-GAN is trained to minimize the difference between pairs of the scrambled images and real images and thus determine find the mapping applied for image scrambling. The adaptation network is essential to extract semantic features from the scrambled images.
B. GENERATOR 1) ADAPTATION NETWORK Figure 4 illustrates the architecture of the adaptation network, which consists of block-wise subnetworks, In block decomposition, scrambled image x e is segmented into N blocks of B × B pixels, {x e 1 , x e 2 , · · · , x e N }, where x e b represents a block (i.e., segmented image). Each block is transformed by the corresponding block-wise subnetwork, f (x e b ; θ b ). Then, the adaptation network is individually trained on each block to handle images processed by block-wise scrambling with different keys. Subsequently, the extracted features are combined on block composition. To input the features to the generator, the combined features are reshaped. In this study, we used a pixel shuffling layer [22] for reshaping.

2) FEATURE DECODER
We propose feature decoder θ fd to convert features { x 1 , . . . , x N } into real images {x 1 , . . . ,x N }. During training, the feature decoder aims to recover the original state from the scrambled images. Table 5 lists the architecture details of the feature decoder. We adapt spectral normalization for GAN [23] in the proposed feature decoder to generate high-quality images.

C. DISCRIMINATOR
We introduce discriminator θ d to determine whether input images are real or synthetic. During training, the discriminator guides the generator to produce more realistic images. Table 6 lists the architecture details of the discriminator.
Again, we adapt spectral normalization for GAN [23] in the proposed discriminator for consistency with the generator.

D. TRAINING
During training, we jointly train the discriminator and generator, whose adaptation network and feature decoder are simultaneously updated.
Scrambled image { x 1 , . . . , x N } is input to adaptation network θ a for feature extraction. Output { x 1 , . . . , x N } with rich information about the original image is fed to the generator to recover the original state of the scrambled image, {x 1 , . . . ,x N }. Finally, output {G(x 1 ), . . . , G(x N )} is fed to the discriminator to learn the mapping onto the real image.
The loss function for training the adaptation network, generator, and discriminator is given by where G denotes the generator, D denotes the discriminator,Ĝ denotes the generator with parameter freezing,D denotes the discriminator with parameter freezing, and L adv denotes the adversarial loss in spectral normalization for GAN [23]. The adversarial loss is aimed at transforming the scrambled image into the original real image through adversarial training. Adversarial loss L adv (G, D) is given by where x denotes the real image,x denotes the scrambled image, and G(x) denotes the generator output. Using Eq. 2, the parameters of generator and discriminator are updated alternately. In this study, we used the hinge loss to compute the adversarial loss. The adversarial loss of the discriminator is computed as where L adv (Ĝ, D) denotes the update of the discriminator with parameter freezing of the adaptation network and generator. The adversarial loss of the generator is computed as where L adv (G,D) denotes the update of the adaptation network and generator with parameter freezing of the discriminator. This adversarial loss aims at updating the generator parameters.

E. ATTACK
During the application of SIA-GAN, we assumed that the image scrambling method from the test data was the same as that from the training data. The inference operation is given by where x denotes the generator output, G denotes the generator, andx denotes the scrambled image in test data.

V. EXPERIMENTAL VALIDATION
The performance of SIA-GAN was evaluated using the adaptation network to confirm the effectiveness of feature extraction. The CIFAR-10 and CIFAR-100 datasets were used for both qualitative and qualitative evaluations. For the qualitative evaluation, images were generated by SIA-GAN to understand the attack results regarding human perception. For the quantitative evaluation, we used the reliable LPIPS score [24]. LPIPS was recently proposed to evaluate the perceptual similarity of generated images and original images. We evaluated original image and scrambled images obtained by applying PE, random PE, LE, ELE, and EtC (Table 1). The images were converted into block-wise or pixelwise scrambled images to be used as inputs to the adaptation network. Data augmentation was applied before block-wise or pixelwise scrambling.
For evaluation, we used the adaptation network, generator, and discriminator as detailed in Tables 4, 5, 6, respectively. In adaptation network, the kernel size was adjusted according to the scrambled images. For PE and random PE, the kernel size was 1 × 1 because these methods use pixel-wise scrambling operations. For LE, ELE, and EtC, the kernel size was 4 × 4 given the block size of 4 × 4 pixels. Architecture of feature decoder. Spectral normalization is conducted before batch normalization, which is followed by activation. TABLE 6. Architecture of discriminator. Spectral normalization is applied before batch normalization, which is followed by activation.

A. SIA-GAN APPLIED TO SCRAMBLED IMAGES 1) EXPERIMENTAL SETUP
The minibatch size was set to 64 during training and testing. Adam optimizer was applied with beta1 = 0.0, beta2 = 0.9. A learning rate of both the adaptation network, and generator is set to 1e-4, and discriminator is set to 4e-4. The training model was updated after every iteration, the network was trained for 100 epochs. Although GAN training generally requires several epochs, 100 epochs provided satisfactory results to verify the effectiveness of the proposed SIA-GAN in the evaluations. The 50,000 images in the CIFAR-10 dataset were used for training the model. All training images were converted into scrambled images to resemble an attacker crawling those images. For the real images, we considered the CIFAR-10 and CIFAR-100 datasets to represent a scenario in which the attacker does not know the original dataset although similar datasets can be collected. Table 2 lists the images generated by the proposed SIA-GAN. If block shuffling is applied, as in ELE or EtC, the scrambled imagescannot be converted back into the original images. Therefore, block shuffling seems suitable to protect scrambled images from GAN-based attacks.

2) EXPERIMENTAL RESULTS
For the other types of scrambled images, SIA-GAN recovered the original images from the scrambled images. Although the quality of PE and random PE aren't good enough, the structure of content can be confirmed. Although the quality of images recovered after the application of PE and random PE is poor, the structure of their contents can be inferred. Although random PE applies scrambling with random parameters, SIA-GAN can suitably reconstruct the original state. Regarding the real images, we confirmed that the attack results are similar, indicating that using natural images is important to attack scrambled images.
Overall, we experimentally confirmed that block shuffling in the scrambled image is important in image scrambling to achieve robust protection against model-inversion attacks that minimize the classification error. VOLUME 9, 2021 Table 3 lists the image quality of scrambled images and recovered images in term of LPIPS. If the proposed attack can reconstruct the original state, the corresponding LIPIS score is high.
These results indicate that SIA-GAN can reconstruct the original state if block shuffling is not applied for scrambling. The LPIPS scores quantitatively show reasonable reconstruction when applying the proposed SIA-GAN.

VI. CONCLUSION
We propose SIA-GAN to evaluate the privacy protection level provided by various types of image scrambling methods. SIA-GAN integrates an adaptation network into a conventional GAN to extract visual information from scrambled images. This type of attack processes scrambled images using model inversion. For SIA-GAN, we consider more advantageous settings for malicious attacks than using real images whose domain is the same as that of scrambled images. Indeed, we consider pairs of real images and scrambled images. Then, the GAN minimizes the distribution difference between scrambled images and real images to learn the image scrambling operations. Experimental results indicate that block shuffling effectively protects scrambled images from the GAN-based attack. In practice, block shuffling degrades the classification performance, as described in our previous work [6]. For privacy-preserving machine learning, block shuffling should be applied during image scrambling.

APPENDIX MODEL ARCHITERCURE
See Figure 5 and Tables 4-6