Evaluating Differentially Private Generative Adversarial Networks Over Membership Inference Attack

As communication technology advances with 5G, the amount of data accumulated online is explosively increasing. From these data, valuable results are being created through data analysis technologies. Among them, artificial intelligence (AI) has shown remarkable performances in various fields and is emerging as an innovative technology. In particular, machine learning and deep learning models are evolving rapidly and are being widely deployed in practical applications. Meanwhile, behind the widespread use of these models, privacy concerns have been continuously raised. In addition, as substantial privacy invasion attacks against machine learning and deep learning models have been proposed, the importance of research on privacy-preserving AI is being emphasized. Accordingly, in the field of differential privacy, which has become a de facto standard for preserving privacy, various mechanisms have been proposed to preserve the privacy of AI models. However, it is unclear how to calibrate appropriate privacy parameters, taking into account the trade-off between a model’s utility and data privacy. Moreover, there is a lack of research that analyzes the relationship between the degree of differential privacy guarantee and privacy invasion attacks. In this paper, we investigate the resistance of differentially private AI models to substantial privacy invasion attacks according to the degree of privacy guarantee, and analyze how privacy parameters should be set to prevent the attacks while preserving the utility of the models. Specifically, we focus on generative adversarial networks (GAN), which is one of the most sophisticated AI models, and on the membership inference attack, which is the most fundamental privacy invasion attack. In the experimental evaluation, by quantifying the effectiveness of the attack based on the degree of privacy guarantee, we show that differential privacy can simultaneously preserve data privacy and the utility of models with moderate privacy budgets.


I. INTRODUCTION
With the development of 5G communication technology that diversifies the access environment and materializes distributed networks, various types and vast amounts of data are being accumulated online. From these data, valuable results are being created through data analysis technologies.
The associate editor coordinating the review of this manuscript and approving it for publication was S. K. Hafizul Islam .
In particular, machine learning and deep learning technologies have been widely used and have shown remarkable performances in various areas such as classification, language representation, recommendations, synthetic data generation, etc. (e.g., [1]- [3]). Moreover, with the introduction of machine learning as a service (MLaaS), which is a range of machine learning functionality offered by cloud service providers, the use of artificial intelligence (AI) models is becoming more active. Typically, these models are generated by learning massive amounts of raw data, and this can lead to revealing sensitive individual information.
Indeed, along with the widespread deployment of artificial intelligence models, concerns about privacy violations have been raised. In addition, as substantial privacy invasion attacks on AI models have been proposed recently [4]- [10], the importance of research on privacy-preserving AI has been emphasized. Accordingly, various approaches have been introduced to preserve the privacy of AI models. Among them, differential privacy [11], [12], which has become a de facto privacy standard, provides a rigorous privacy guarantee, and various mechanisms that satisfy the properties of differential privacy have been proposed for designing privacypreserving AI.
Generally, differentially private mechanisms return noisy outputs that obscure statistical differences between adjacent databases, and the magnitude of the noise that will be added to the actual output for a specific query is highly dependent on the privacy parameter , called the privacy budget. In other words, the lower the privacy budget, the larger the noise, and vice versa. Obviously, from the perspective of utility as well as privacy, the choice of is one of the most important factors, and should be calibrated with in-depth consideration of the trade-off between privacy and utility. However, the criterion for how to set an appropriate privacy budget has not been clearly established in practice, and then differentially private AI models have often set the privacy budget as a tendency to ensure acceptable utility. As a result, the utility of the models may be able to be guaranteed, but privacy may not be preserved at all. This ambiguity is a well-known problem in the field of differential privacy, and we aim to address this problem by analyzing the relationship between differential privacy and substantial privacy invasion attacks for AI models.
In this paper, we evaluate the resistance of differentially private AI models to substantial privacy invasion attacks by varying the privacy budget , and analyze how privacy parameters should be set to prevent the attacks while preserving the utility of the models. Furthermore, we study the efficacy of privacy invasion attacks under the relaxed notions of differential privacy (i.e., concentrated differential privacy [13], zero concentrated differential privacy [14], and Rényi differential privacy [15]), and analyze how much of a privacy breach occurs by relaxing the definition of differential privacy on AI models. In particular, we focus on the generative model, especially generative adversarial networks (GAN), which is one of the most sophisticated models for generating synthetic datasets and has attracted great interest recently. In the case of attack scenario, we focus on the membership inference attack, which is the most fundamental privacy invasion attack.
Currently, several results have been reported for evaluating differentially private AI models under substantial privacy invasion attacks [16]- [20]. However, these results have mainly focused on neural network-based models or regression models, and there are no results for models with the objective of generating synthetic dataset. In this respect, we aim to contribute to the evaluation of differentially private mechanisms for generative models by analyzing the relationship between the degree of differential privacy guarantee and the privacy invasion attack.
The rest of this paper is organized as follows. First, there is a background review in section 2. Then, we present differentially private mechanisms and the membership inference attack for GAN models in section 3. In section 4, we describe our evaluation framework, and demonstrate experiment and evaluation results in detail. Finally, we discuss related studies in section 5, and then conclude our work in section 6.

II. BACKGROUND
In this section, we briefly illustrate generative adversarial networks (GAN), and review the definition of differential privacy and its relaxations. Then, we demonstrate mechanisms that can make GAN models differentially private.

A. GENERATIVE ADVERSARIAL NETWORKS
Generative models are designed for learning the probability distribution of a given training data, and have the purpose of generating synthetic data close to the real data. Among the various generative models, great interest has been focused on generative adversarial networks (GAN) [21], and numerous studies have been conducted to advance their performance and functionality. As shown in Figure 1, the basic architecture of GAN consists of two neural network-based components: a generator G and discriminator D. The generator G takes noise z (latent code) as an input and generates synthetic data x with the objective of generating data that approximates the real data x while the discriminator D takes a dataset consisting of the synthetic data and real data with the objective of discriminating the difference between real (training data x) and fake (synthetic data x ). Therefore, these two components always play a game to beat each other, and are trained alternately.
More formally, let p z be the probability distribution of the latent code and p data be the probability distribution of the real data. Then the objective function V (D, G) of the GAN model that consists of G and D is a minimax game, and can VOLUME 9, 2021 be formulated as follows.
where θ D and θ G denote the parameters of the discriminator and generator, respectively. Therefore, the discriminator is trained to return a high score in a given training samples (real data), and the generator is trained to produce synthetic data that can maximize the discriminator's output. After sufficient training, if a Nash equilibrium is achieved, both the discriminator and generator settle at a point where there is no further improvement.
Since the basic concept of GAN was introduced, numerous variants have been proposed with the aim of evolving the original model by adjusting the objective function or by modifying the architecture (e.g., [22]- [27]). Among these variants, we target several significant models that have shown noticeable improvement: 1) deep convolutional GAN (DCGAN) [23], a model that combines the basic GAN architecture with a convolutional neural network. 2) Wasserstein GAN (WGAN) [25], a model that improves training stability by using the Wasserstein distance (instead of Jensen-Shannon divergence in the original GAN model) as an approximation metric between probability distributions. 3) boundary equilibrium GAN (BEGAN) [26], a model that can approximate the convergence of the training process by combining with the concept of Autoencoder.

B. PRIVACY INVASION ATTACKS ON AI MODELS
Along with the advancement of AI technology, privacy concerns have been raised simultaneously, and various attacks that can substantially invade privacy on machine learning and deep learning models have been proposed. Ateniese et al. [28] showed that it is possible to infer the general statistical information about a training dataset by exploiting the internal parameters of specific models (such as Support Vector Machines and Hidden Markov Models). For a collaborative recommender system, Calandrino et al. [29] reported that, by capturing changes between outputs, an attacker can infer specific inputs that triggered those changes. As an attack that can directly invade the privacy of training data, Fredrickson et al. [4], [5] proposed the model inversion attack, where an attacker can reconstruct parts of information in the training dataset by exploiting confidence vectors returned along with predictions from a target model. Tramèr et al. [7] introduced the model extraction attack that can extract parameters of the target model, and showed that sensitive information of the training dataset can be exposed. As a fundamental privacy invasion attack, Shokri et al. [6] proposed the membership inference attack, where an attacker can infer whether or not specific input data were included in the training dataset. The underlying intuition of the attack is that a machine learning model (trained model) will behave differently between the data that the model has learned (i.e., training data) and unseen data, and these differences become more severe when the model is overfitted to the training data. In order to construct a (membership inference) attack model, an adversary has to build multiple shadow models that mimic the target model, and these shadow models can be built by learning the confidence vectors obtained from the target model (by querying the target model with arbitrary inputs). Then, the attack model can be constructed by learning the results (confidence vectors) output from the shadow models for both data with and without membership. Note that, in the case of the shadow model, an attacker can know exactly whether a given data was included in the training dataset of the shadow model.
Since the concept of membership inference attack against general machine learning models was introduced, various studies have been conducted to analyze and advance the attack [8]- [10], [30]. In particular, attack methods have been proposed that focus on generative models with a different aspect from previous studies that targeted general machine learning models. Since the outputs of a generative model are synthetic data rather than predictions, it is necessary to consider a different approach from the previous methods. By capturing these points, Hayes et al. [8] proposed an attack method with the goal of membership inference against GAN models. Subsequently, inspired by Hayes et al.'s study, several attack methods have been proposed with different assumptions and attack scenarios [9], [10] in terms of distance metric. We focus on these attacks as our goal is to analyze the relationship between substantial privacy invasion attacks and privacy-preserving techniques over GAN models (a detailed analysis of each membership inference attack against GAN is covered in section 3).

C. DIFFERENTIAL PRIVACY
Differential privacy [11], [12] has become a de facto privacy standard and ensures strong privacy preservation. Intuitively, a mechanism that satisfies differential privacy returns similar outputs on adjacent datasets for a given query. It means that differentially private mechanisms provide plausible deniability against adversaries.
If two datasets d, d ∈ D differ in one entry, we say that the datasets are adjacent (or neighboring). Then, the definition of differential privacy is as follows.
Definition 1 (( , δ)-Differential Privacy (( , δ)-DP) [11], [12]): A randomized mechanism M satisfies ( , δ)-differential privacy, for all output S ⊆ Range(M) for any two adjacent datasets d, d , if we have: Obviously, a smaller value of privacy cost (privacy budget) leads to a better privacy guarantee, and the value of δ is generally set to be smaller than the inverse of any polynomial in the size of database. When the additive term δ is zero, it is called -differential privacy (pure differential privacy).
In general, differential privacy can be achieved by adding noise to the actual output for a query function, and the magnitude of noise is estimated depending on the sensitivity of the query function.
Definition 2 (l 2 -Sensitivity): For any two adjacent datasets d, d , the sensitivity of query function f is defined as follows: There are several basic mechanisms that ensure differential privacy, including Laplace mechanism and Gaussian mechanism, and we focus on the Gaussian mechanism, which has been widely leveraged to achieve differential privacy for AI models.
Definition 3 (Gaussian Mechanism): For any dataset d and query function f (·), the Gaussian mechanism M G is defined as follows: where N (0, σ 2 ) is the Gaussian distribution with mean 0 and standard deviation σ .
Note that a single execution of the Gaussian mechanism satisfies ( , δ)-DP if σ > 2 ln 1.25/δ · f / and < 1. Basically, in terms of running multiple times, differential privacy provides the composition property. If two mechanisms M 1 and M 2 satisfy 1 and 2 -DP respectively, then a family of the mechanisms M = (M 1 , M 2 ) satisfies ( 1 + 2 )-DP.
Since the definition of differential privacy was introduced, several notions that can relax the original definition of differential privacy have been proposed to analyze a tighter bound in terms of cumulative privacy loss over multiple executions by considering the fact that the privacy loss random variable is tightly concentrated around its expectation. There are three commonly used relaxed definitions of differential privacy: concentrated differential privacy [13], zero concentrated differential privacy [14], Rényi differential privacy [15]. 1 In a subsequent study of pure and ( , δ)-differential privacy, Dwork et al. [13] introduced concentrated differential privacy (CDP) by focusing on the case where the privacy loss follows a sub-Gaussian distribution. The intuition embedded in the notion of CDP is that the privacy loss is strictly centered around its expectation and the tail is managed by the variance of the sub-Gaussian distribution.
Definition 4 (Concentrated Differential Privacy (CDP) [13]): A randomized algorithm M is (µ, τ )-concentrated differentially private if, for all pairs of adjacent databases d, d , we have: where D subG denotes the sub-Gaussian divergence.
The definition means that the expected privacy loss is bounded by µ and the distribution of the centered privacy loss (by abstracting µ) is sub-Gaussian with standard deviation τ . In terms of relevance to the previous notion, the authors showed that if a mechanism M satisfies -DP algorithm, then M ensures ( ·(e −1)/2, ))-CDP (but the converse does not hold). In addition, they showed that the Gaussian mechanism defined above satisfies (τ 2 /2, τ )-CDP with τ = f /σ .
In a subsequent study on CDP, Bun et al. [14] proposed the notion of zero-concentrated differential privacy (zCDP). By reformulating the concept of CDP through the Rényi divergence, they analyzed a tighter bound on the cumulative privacy loss over multiple computations.
Definition 5 (Zero-Concentrated Differential Privacy (zCDP) [14]): A randomized mechanism M is (ξ, ρ)-zero-concentrated differentially private if, for all adjacent databases d, d and all α ∈ (1, ∞), we have: ) denotes the α-th moment's Rényi divergence between the distribution M(d) and the distribution M(d ). 2 zCDP can be directly related to previous definitions of differential privacy through the Rényi divergence. In [14], the authors showed that if a mechanism M satisfies -DP, then M is ( 1 2 2 )-zCDP, and moreover proved that if a mechanism M ensures ρ-zCDP, then M satisfies (ρ + 2 √ ρ log(1/δ), δ)-DP. Furthermore, they showed that the definition of CDP and zCDP can be interpreted mutually. In the case of the Gaussian mechanism, they proved that the mechanism (in Definition 3) satisfies ( f 2 /2σ 2 )-zCDP.
Based on the Rényi divergence, the notion of Rényi differential privacy was introduced as a natural relaxation of differential privacy [15], where the definition of differential privacy is relaxed by bounding the Rényi divergence of the privacy loss random variable for any individual moment.
Definition 6 (Rényi Differential Privacy (RDP) [15]): A randomized mechanism M is said to have -Rényi differential privacy of order α (or (α, )-RDP for short), if for any adjacent databases d, d it holds that: Different from the other definitions, RDP bounds the Rényi divergence of privacy loss random variable only for a single moment at a time, as shown in the definition, which allows the analysis of a tighter bound on the cumulative privacy loss. In [15], the authors showed that if a mechanism M satisfies (α, )-RDP, it also satisfies ( + log 1/δ α−1 , δ)-DP for any 0 < δ < 1. Furthermore, they showed that the Gaussian mechanism M G satisfies (α, α f 2 2σ 2 )-RDP.

III. MEMBERSHIP INFERENCE ATTACK AGAINST DIFFERENTIALLY PRIVATE GENERATIVE MODEL
In this section, we describe each major part in detail. First, we analyze membership inference attacks and explain how to achieve differential privacy for the GAN model. Then we present our evaluation framework for analyzing the relationship between the privacy invasion attacks and differentially private GAN models.

A. MEMBERSHIP INFERENCE ATTACK AGAINST GENERATIVE MODEL
As mentioned above, previous membership inference attacks (e.g., [6], [30]) targeted general machine learning models and aimed to infer whether specific data were included in the training dataset by exploiting the confidence vectors returned from the target model. Unlike general machine learning models consisting only of discriminative components, GAN consists of two distinct components (discriminator and generator) with opposite objectives, and it is regarded that the final result of training GAN is the generative part. Therefore, since the outputs of GAN for arbitrary inputs (latent codes) are synthetic data rather than confidence values, it makes no sense to apply previous membership inference attacks against GAN model. By capturing these differences, Hayes et al. [8] showed that it is feasible to implement membership inference attacks against generative models. The intuition involved in this approach is that since the discriminator tends to output (even slightly) higher probability in training data than in others (synthetic data and testing data), the discriminator can be exploited as a distinguisher for membership inference attacks. Therefore, as long as an attacker builds a shadow discriminator that mimics the discriminator of the target model, he/she can execute membership inference attacks without building an additional attack model. Note that from the attacker's point of view, it is not allowed to access the discriminator of the target model, 3 and we assume that the white-box attack where an attacker is allowed to access the discriminator is the ideal scenario.
To build a shadow discriminator, an attacker first collects enough data (synthetic data) via queries to the target generative model, and then trains a new GAN model using the data obtained. At this point, the discriminator of the GAN model constructed by the attacker becomes an attack engine (shadow discriminator) that drives membership inference. After building a shadow discriminator, the attacker can conduct membership inference on data that the attacker holds and wants to know about membership (not the data obtained by querying the target model) by feeding the data to the shadow discriminator. The final inferences for specific data are made by arranging the data according to the results of the shadow discriminator. That is, the attacker can determine that data was included in the training dataset of the target generative model if the data is ranked at the upper position (relatively high probability) in the results of the shadow discriminator. In [8], the authors showed that the ideal white-box attacker who can access the discriminator of the target model can perfectly infer membership (100% attack accuracy). In the case 3 In general, it is considered that GAN model is a generative model, and the discriminator that was operated when training GAN is not publicly exposed. of a black-box scenario, they showed that a general attacker can achieve up to 63% attack accuracy, and an informed attacker who has auxiliary knowledge of the training dataset can improve the attack success rate (these outcomes are the experimental results on a facial dataset, and we deal with the same dataset as our experimental data in section 5).
As an extended approach of the attack against the generative model, Hilprecht et al. [9] proposed a Monte Carlo integration-based membership inference attack in terms of distance metric between training and generated data. The intuition of this approach is that the overfitted generator tends to output synthetic data close to the training data that the target model has learned. In [9], the authors considered blackbox scenarios and showed that an attacker can conduct membership inference on specific data in the Monte Carlo method by comparing the distance with synthetic data obtained from the target model.
From the perspective of distance metric, Chen et al. [10] proposed a more sophisticated method of attack by subdividing the attacker's knowledge about the generator of the target GAN model. They assumed black-box and white-box scenarios for the generator, and considered that the adversary can access the latent code that is the input of the model. With these assumptions, they realized a more sophisticated distance-based membership inference attack by reconstructing the synthetic data as close as possible to the target data that the attacker wants to know about membership. Note that in the case of the white-box scenario for the generator, an attacker can obtain data closer to the target data through an optimization process on the input space of the generator (e.g., gradient descent).
From the perspective of analyzing the relationship between privacy and utility for the GAN model, we deploy these membership inference attacks as privacy violation scenarios. In particular, we consider two specific attack scenarios (see Figure 2): ideal white-box (i.e., accessible to the discriminator) and realistic white-box (i.e., white-box for the generator with latent code) scenarios. For the ideal white-box scenario, the discriminator of the target model will be operated as the attack engine. In the case of the realistic white-box scenario, we assume that an adversary can access the parameters of the generator as well as latent code.

B. DIFFERENTIAL PRIVACY FOR GENERATIVE ADVERSARIAL NETWORKS
Since the notion of differential privacy emerged, extensive studies have been conducted in various fields requiring privacy preservation. Obviously, in the field of artificial intelligence, research on differentially private mechanisms has been actively conducted to preserve the privacy of AI models. Initially, the results were mainly focused on convex optimization problems and general machine learning models such as ERM [31], [32], decision tree [33], [34], regression [35], [36] etc., and progressed toward satisfying differential privacy for complex models with non-convex optimization problems such as deep learning and autoencoder (e.g., [37]- [39]).
There are three main approaches to achieve differential privacy for complex AI models: output perturbation, objective perturbation, and gradient perturbation. Among these approaches, most mechanisms that satisfy differential privacy for the GAN model have leveraged the gradient perturbation approach because of its flexibility and adaptability. Note that output perturbation can introduce a huge amount of noise in the parameters of the final model as differential privacy pursues the worst-case scenario, and objective perturbation is not generally applicable as it depends on the architecture of the model. The gradient perturbation method satisfies differential privacy by adding noise in the learning process of the model that performs gradient-based optimization. That is, at each iteration of the training, noise is added to the gradients calculated by referring to the dataset that contains sensitive information so that differential privacy can be held. The main issue with this approach is how to calculate tighter bounds on privacy loss in terms of composition. In this respect, Abadi et al. [37] recently proposed an efficient differentially private learning algorithm. In [37], the authors presented the moments accountant mechanism that can efficiently bound the cumulative privacy loss of the algorithm, and showed that differential privacy can be achieved with a modest privacy budget while preserving the utility of model. Most of the differentially private GAN algorithms leverage the moments accountant to calculate the cumulative privacy loss. Table 1 presents the characteristics for each differentially private GAN algorithm.
Xie et al. [40] proposed a differentially private GAN algorithm at first. They focused on the Wasserstein GAN model, and showed that differential privacy can be achieved via the same gradient-based training process as deep learning models. In principle, since GAN models consist of a discriminator and generator, unlike deep learning models that consist only of discriminative ones, it should be considered that differential privacy has to be applied to two distinct sub-models. However, Xie et al. [40] showed that if differential privacy is involved in the training of the discriminative model that directly references the training dataset, the generative model also satisfies differential privacy naturally due to the post-processing immunity of differential privacy. In the case of composition, they leveraged the moments Compute gradient for discriminator 7: Clip gradient 10:ḡ w t ,real (x (i) ) ← g w t ,real (x (i) )/ max (1, for each x (i) ∈ B t 11:ḡ w t ,fake (z (i) ) ← g w t ,fake (z (i) )/ max (1, Add noise 13:g w t ← 1 m ( iḡw t ,real (x (i) ) + N (0, σ 2 c 2 I)) − 1 m iḡw t ,fake (z (i) ) 14: Optimize discriminator 15: w t+1 ← OPT(w t ,g w t , η D ) 16: ..,m ∼ P z of latent code 18: θ t+1 ← OPT(θ t , g θ t , η G ) 20: end for Output: model parameters of discriminator w T and generator θ T , and compute the overall privacy cost ( , δ) accountant theorem. At around the same time, Srivastava and Alzantot [41] proposed a differentially private WGAN algorithm for the purpose of privacy-preserving synthetic data generation. The difference from the previous algorithm is that they simply bounded the sensitivity of gradients with the gradient clipping method. Although Xie et al. [40] approximated the upper bound of gradients, it can cause greater noise levels than the simple clipping approach as the approximation depends on the size of the model. Subsequently, several results have been reported that extend the previous algorithms to the improved WGAN model (WGAN-GP [42]) [43]- [45]. As another result on the differentially private GAN model, Torkzadehmahani et al. [46] proposed a differentially private conditional GAN algorithm that can generate both differentially private synthetic data and corresponding labels by utilizing the characteristics of the conditional GAN model. In particular, by applying Rényi differential privacy, they showed that the algorithm can improve the quality of synthetic data in the same privacy budget compared to algorithms involving the basic notion of differential privacy. For the purpose of privacy-preserving data sharing for clinical data, Beaulieu-Jones et al. [47] applied differential privacy to the auxiliary classifier GAN (AC-GAN [48]). As with the previous algorithms, they leveraged the differentially private SGD algorithm and calculated the overall privacy loss with the moments accountant.
By extending the capacity of these algorithms, we evaluate the impact of differential privacy regarding the privacy budget and its relaxed definitions over the substantial privacy violation scenario. 4 Algorithm 1 presents a systematic algorithm for achieving differential privacy in the training of GAN models. First, the algorithm samples a mini-batch with the sampling probability q = m n from the (training) dataset, and a mini-batch of size m from the latent space randomly. Then the gradients of the loss function L D for the discriminator are computed with respect to the current parameters w t of discriminator in both mini-batches, and the computed gradients are clipped by l 2 -clipping with the clipping parameter c. At this point, Gaussian noise is added to the summed gradient to ensure differential privacy, and the magnitude of the noise is derived by considering only the gradient associated with the real data. After the gradient is averaged and aggregated, the parameter w t of the discriminator is updated with the calculated gradientg w t and learning rate η D in a gradient-based optimization method, such as SGD, Adam, or RMSProp. Note that the optimization process may include adjustments to model parameters, such 4 We do not cover strategic approaches [49]- [51] (for example, clipping decay to reduce the magnitude of noise) because we focus on analyzing the relationship between differential privacy and the privacy invasion attack according to the privacy budget and the relaxed definitions of differential privacy TABLE 2. Comparison of relaxed definitions of differential privacy. The DP interpretation of Concentrated DP is derived indirectly via zCDP [14].
as weight clipping to ensure Lipschitz continuity in WGAN. This discriminator training procedure (lines 3 ∼ 15 in the algorithm) can be iterated in several steps internally, as in the case of WGAN. Obviously, in this case, privacy costs arise in every sub-iteration. After completing the training of the discriminator in an iteration step t, the algorithm trains the generator G, and this procedure is the same as regular generator training in the non-private scenario. Since the procedure for training G (lines 16 ∼ 19 in the algorithm) is a postprocessing of a differentially private discriminator and does not access the training dataset, there is no need to force this procedure to ensure differential privacy. When the algorithm is finished, it outputs the final model parameters w T and θ T , and computes the overall privacy cost spent. We consider differential privacy and its relaxations, and Table 2 compares the noise scale in a single execution of the Gaussian mechanism according to the definitions of differential privacy. Although Algorithm 1 computes the overall privacy cost as an output process, it can be calculated during the running of the algorithm (right after optimizing the discriminator each time). In this case, a predefined entire privacy budget can be specified as termination criteria (i.e., as a threshold for spent privacy costs).
To evaluate differentially private GAN models, we first generate models via Algorithm 1, and then analyze their resistance over the membership inference attack scenarios according to the degree of the privacy guarantee and the definitions of differential privacy. Note that, the algorithm presents the process of training GAN models to satisfy differential privacy, and makes no assumptions about specific attack scenarios.

IV. EVALUATION
In this section, we conduct experiments to quantify how much privacy is leaked from differentially private GAN models. As mentioned in the previous section, we measure privacy leakage via membership inference attack in ideal and realistic adversarial scenarios.

A. EXPERIMENTAL SETUP
We first train target GAN models using Algorithm 1 with different relaxed notions of differential privacy, and compare them in terms of privacy leakage. The notions that we consider are ( , δ)-DP, zero-concentrated DP (zCDP), and Rényi DP (RDP). Since concentrated DP (CDP) has the same composition property and noise scale as zCDP, as shown in Table 2, we do not include CDP in the experiments. For zCDP, we convert privacy budgets to ( , δ)-DP, and use them as termination thresholds. In the case of RDP, we leveraged the RDP accountant [52], [53].
As described above, relaxing the definition of differential privacy results in a smaller noise scale for a given privacy budget. Alternatively, in terms of composition, the relaxed notions enable more differentially private operations for a given privacy budget and fixed noise scale. In this respect, we considered the latter case and set the noise scale σ = 2. With respect to the sensitivity in terms of differential privacy, we set the gradient clipping parameter c = 2. Note that, since the parameters c and σ are directly involved in the standard deviation of the Gaussian distribution in the Gaussian mechanism, large c and σ can cause large noise, and in this case, the training of GAN models may not proceed at all even with a large number of iterations.

1) TARGET MODEL
We experiment and evaluate three GAN models: 1) deep convolutional GAN (DCGAN), a model that combines the basic GAN architecture with a convolutional neural network, 2) Wasserstein GAN (WGAN), a model that improves training stability by using the Wasserstein distance as an approximation metric between probability distributions, 3) boundary equilibrium GAN (BEGAN), a model that can approximate the convergence of the training process by combining it with the concept of Autoencoder.
For DCGAN and WGAN, we built both models with the same architecture. In particular, we constructed the discriminator as three convolutional layers and a fully connected layer sequentially. Note that, since the differentially private learning algorithm computes the gradient for each single data point, we did not include the batch normalization process due to compatibility concerns. For the generator, we consisted of a fully connected layer and three de-convolutional layers (upsampling-convolutional layer) sequentially. In the case of BEGAN, we constructed the encoder as three convolutional layers and a fully connected layer, and the decoder as a fully connected layer and three de-convolutional layers (the discriminator is constructed by encoder-decoder and the architecture of the generator is the same as that of the decoder). Unlike the other models, WGAN internally iterates discriminator training before proceeding with generator training, and we set the number of internal iterations to 5.

2) DATASET
We use two datasets to evaluate differentially private GAN models: the MNIST handwritten dataset containing 60k training samples and 10k testing samples of size 28 × 28 in grayscale, and the Labeled Faces in the Wild (LFW) dataset [54] containing 13,233 images of faces. In the case of LFW datset, we aligned each data to a size of 62 × 47 and converted it to grayscale. For both datasets, we randomly sample 10% of the data points as the training dataset. To measure the privacy leakage, we also prepare a test dataset (for attack scenario) with the same manner and size as the training dataset. From the perspective of the attack scenario, the data points in the training dataset are members, and the others are non-members. Therefore, the attack success rate of the baseline attacker can be 50%.

B. MODEL ACCURACY
Before evaluating the resistance of differentially private GAN models to the membership inference attack, we investigate the accuracy of the models according to the notions of differential privacy as well as privacy budgets in terms of the quality of generated synthetic data. Figure 3 presents generated samples from trained differentially private GAN models. Obviously, it can be seen that the quality of the generated data improves as the privacy budget increases. Likewise, in the same privacy budget, the quality of the generated data is improved as the notion of differential privacy is relaxed. To show the results from a broader perspective, we further present the outputs of differentially private GAN models trained on the CelebA dataset [55] (under the same conditions as the experiments in the MNIST and LFW datasets), which is an RGB threecolor (celebrity) face dataset. Compared to other models, (differentially private) BEGAN seems to outperform even with relatively small privacy budgets.
In addition to these visual comparisons, we conduct a classification task on the generated data to evaluate the models' accuracy numerically. Although there are several metrics that can evaluate the quality of synthetic data generated from GAN models, such as inception score and SSIM (structural similarity), we adopted the classification accuracy to intuitively represent the quality of synthetic data. Note that, this approach has been applied in previous studies on differentially private GANs (e.g., [46], [49]). The process of this experiment is as follows: we first build a classification model that acts as an evaluator for model accuracy using the original training dataset, and generate synthetic datasets from the trained differentially private GAN models with the same proportion and size as the original test dataset.
Then we present the classification accuracy of the evaluator on these synthetic datasets as the accuracy of differentially private GAN models. In order to label the generated data, we trained GAN models by including the class attribute as with the conditional GAN architecture [22]. In the case of the classifier, we build a general two-layer neural network model. Figure 4 shows the experimental results on the MNIST dataset. As shown in the results, we found that not only was the visual quality improved, but also the classification accuracy of the generated data, and it can be interpreted that the differentially private GAN models generate clearer data as the privacy budget is increased and the definition of differential privacy is relaxed. 5 In the case of the nonprivate scenario, the accuracy was measured as 0.931 and 0.952 for DCGAN and BEGAN, respectively. In the case of the differentially private scenario, we confirmed that the RDP models converge very closely to the non-private scenario at ≥ 10 compared to the other notions. When = 10, the accuracy loss in the experiments on DCGAN was measured as 12%, 8%, and 5% in DP, zCDP, and RDP, respectively. Similarly, in the experiments on BEGAN, the accuracy loss was measured as 10%, 7%, and 4%.

C. IDEAL WHITE-BOX ATTACK SCENARIO
As described above, we assume that the attacker in the ideal white-box scenario has access to the discriminator of the trained GAN model, and exploits the discriminator as the attack engine (i.e., the attacker model) for membership inference. Figure 5 and 6 show the privacy leakage due to the membership inference attack on GAN models in the ideal white-box attack. In the case of the inference, we sorted the outputs of the discriminator for the suspect data and summarized the top ranked data, and we set 1/2 as the minimum value of the attack accuracy since the attack success rate of the baseline attacker is equal to the probability of flipping a coin. This means that there is no privacy leakage when the attack accuracy is 1/2. Figure 5 shows the experimental results on the MNIST dataset. In the non-private scenario, the attacker achieved 93%, 93%, and 57% in DCGAN, WGAN, and BEGAN, respectively. As shown in the figure, we found that differential privacy can significantly reduce privacy leakages even with relatively large privacy budgets. In the case of DCGAN and WGAN, it was measured that the higher the privacy budget and the more relaxed the definition of differential privacy, the more vulnerable to attack (see Figure 5 (a) and (b)). As expected, ( , δ)-DP showed the strongest resistance compared to the other notions, and attack accuracy was measured close to the baseline attacker. In the case of zCDP and RDP, the attack resistance was measured to be strong at < 10, but the attack success rate increased at ≥ 10. In the case of BEGAN ( Figure 5 (c)), different results from the experiments with the other models were measured.  As the privacy budget grows, the attack probability seems to increase slightly. However, the attack success rate was measured with a very low probability even with large privacy budgets (the maximum attack success rate was measured to be 0.53 when 10 < < 15), and it was observed to have a strong resistance to the membership inference attack compared to other models. Figure 6 shows the experimental results on the LFW dataset. In the non-private scenario, the attacker achieved 99%, 99%, and 62% in DCGAN, WGAN, and BEGAN, respectively. Compared with the experiments on the MNIST dataset, the attacker in the non-private scenario achieved higher attack success rates. Overall, the results showed a pattern similar to that of the MNIST experiments. In the case of DCGAN and WGAN, it was measured that the attacker's advantage slightly increased compared to the experiment with the MNIST dataset. However, in the case of BEGAN, the attack success rate was measured with a very low probability as in the previous experiments (the maximum attack success rate was measured to be 0.53).

D. REALISTIC WHITE-BOX ATTACK SCENARIO
As described in the previous section, we assume that the attacker in the realistic white-box scenario has access to the generative model of the trained GAN model and latent code, and exploits them as the reconstruction engine of data for membership inference. Therefore, the attacker model in the realistic white-box attack scenario can be regarded as   a reconstruction process through optimization (optimization process on latent codes to generate data as close as possible to the suspicious data in terms of the distance). Figure 7 and 8 show the privacy leakage due to the membership inference attack on GAN models in the realistic white-box attack. As in the previous experiment, we sorted and summarized the distance results between synthetic and suspect data, and we set 1/2 as the minimum value of the attack accuracy and assume that there is no privacy leakage when the attack accuracy is 1/2. In addition, we excluded the experiment with BEGAN from the realistic white-box attack experiment since it showed strong resistance to the attack even in the ideal white-box scenario. Figure 7 shows the experimental results on the MNIST dataset. In the non-private scenario, the attacker achieved 56% and 61% in DCGAN and WGAN, respectively. As shown in the figure, we found that differential privacy can reduce privacy leakages, and attack accuracy was measured very close to the baseline attacker. In both models, differential privacy showed strong resistance to the attack even with large privacy budgets and relaxed definitions, unlike the experimental results in the ideal white-box scenario. Overall, the attack success rate was measured to be less than 0.53 in both experiments. Figure 8 shows the experimental results on the LFW dataset. In the non-private scenario, the attacker achieved 57% and 63% in DCGAN and WGAN, respectively. As expected, the measured experimental results were very similar to the experiment with the MNIST dataset.

E. DISCUSSION
From the perspective of model utility, the tighter bounds on the cumulative privacy loss by the relaxed definition of differential privacy improve the quality of synthetic data for a given privacy budget. However, in the ideal white-box attack scenario, we found that differentially private models with relaxed definitions are more vulnerable to the membership inference attack because they reduce the noise magnitude or allow more training iterations on a given dataset. Therefore, we can conclude that relaxing the definition of differential privacy comes with additional privacy risks. Nevertheless, we confirmed that differential privacy can significantly mitigate the privacy leakage compared to the non-private scenario even with relatively large privacy budgets. In particular, this advantage is more evident in the realistic white-box attack scenario. Furthermore, by experimenting with various GAN models, we found that privacy leakage is dependent on the model architecture, and applying differential privacy can amplify resistance to the membership inference attack.
In our experiment, we trained and built GAN models on subsets, which are datasets sampled with a probability of 10% from the original training datasets. In other words, GAN models can easily overfit as they are trained on very small datasets, and this can make the models very vulnerable to membership inference attacks. In [8]- [10], it has been reported that the sampling probability significantly affects the accuracy of membership inference attack, and the smaller the sampling set (i.e., the more severe the overfitting), the more vulnerable to attack. Considering these points, since differential privacy should consider the worst-case scenario, we focused on the sampling probability of 10%, which was the most reasonably vulnerable case in non-private scenarios. 6

V. RELATED WORK
Recently, various studies have been conducted to analyzed the relationship between differential privacy and privacy invasion attacks on machine learning and deep learning models. Rahman et al. [16] investigated the relationship between differential privacy and the membership inference attack, focusing on neural network-based models. In particular, they analyzed the trade-off between utility and privacy by varying the privacy budget. Focusing on the model inversion attack for regression models, Wang et al. [17] proposed a differentially private regression model. In [17], the authors leveraged the functional mechanism to ensure differential privacy, and showed that the proposed differentially private regression model can provide resistance to the model inversion attack while preserving utility. Zhang et al. [18] considered an obfuscation method that injects noise into the input dataset before training the machine learning model, and showed that the data reconstructed by the model inversion attack (from the model with the obfuscation applied) is more blurred compared to the non-private scenario. Park et al. [20] studied the relationship between differential privacy and the model inversion attack. In particular, they focused on face recognition systems based on neural network-based models, and analyzed the trade-off between utility and privacy according to the degree of privacy guarantee in the model inversion attack scenario. Jayaraman and Evans [19] investigated the relationship beetween definitions of differential privacy and privacy invasion attacks. By focusing on the membership inference and attribute inference attack [6], [30], they analyzed the resistance of differential privacy to the attacks for logistic regression and neural network models. In contrast to previous studies that targeted neural network and regression models, we focus on generative adversarial networks, which are the most sophisticated generative models, and analyze the relationship between differential privacy and membership inference attack on GAN models.

VI. CONCLUSION
In this paper, we investigated the resistance of differentially private GAN models to the membership inference attack according to the degree of privacy guarantee. In the experimental evaluation, by quantifying the effectiveness of the attack based on the degree of privacy guarantee, we showed that differential privacy can reduce the attack success rates of membership inference while preserving the quality of synthetic data. However, by investigating several notions of differential privacy, we found that relaxing the definition of differential privacy comes with additional privacy risks. Nevertheless, we confirmed that differential privacy can significantly mitigate privacy leakage compared to the nonprivate scenario. As a future study, it would be interesting to investigate the privacy leakage on the differentially private algorithms with strategic approaches (e.g., clipping decay).