Empirical Analysis of The Privacy-Utility Trade-off of Classification Algorithms Under Differential Privacy

Numerous modern services utilize machine learning (ML) algorithms to provide personalized and better usability to the users. Oftentimes, such algorithms are trained on the privacy-sensitive data of the users. In order to preserve the privacy of the users, many corporations and government entities have adopted privacy-preserving ML algorithms. To this end, differential privacy (DP) is the de facto standard. DP definition is cryptographically motivated and mathematically rigorous, and it is designed to provide strong protection against powerful adversaries. In most cases, an ML algorithm can be incorporated into the DP framework by carefully injecting randomization(s) into the algorithm pipeline. This inevitably negatively affects the algorithm’s performance, giving rise to privacy-utility trade-off. In this paper, we present an empirical analysis of commonly used classification algorithms, such as logistic regression, support vector machine, and neural network based classifiers, under DP. More specifically, we investigate how the classification accuracy and overall privacy budget (two parameters that are often in tension with each other) vary if privacy parameters and relevant dataset parameters are varied for several real datasets. Our investigation provides insights into the impact of incorporating DP on the classification algorithms, and it can aid in algorithm selection for specific use cases and privacy requirements in real-world applications. Additionally, our study highlights the potential for DP algorithms to achieve comparable performance to their non-private counterparts under certain parameter regimes.