Differentially Private Variance Reduced Stochastic Gradient Descent

In this paper, we propose a differentially private stochastic variance reduced gradient algorithm, called DP-SVRG. To privatize SVRG algorithm, we randomize the gradient computation process by injecting random noise.There are two main challenges in this approach: (i) high variance of stochastic gradient updates, and (ii) low per-iteration privacy budget. To cope with these challenges, we employ two advanced techniques recently introduced in the literature. First, we apply control variate technique to stochastic gradient update, shown to effective inreducing the variance. Second, we use the tight composition theorem of zero-concentrated differential privacy (zCDP) toeffectively bound the cumulative privacy cost.To show the effectiveness of the proposed algorithm, we conduct empirical evaluations on a set of real datasets and compare our algorithm with existing approaches.