Differential Privacy Adaptive Gradient Clipping Method Based on Hessian Matrix

With the rapid development of big data technology, sensitive data leakage has brought great privacy risks, which has become one of the severe challenges faced by both academia and industry. Differential privacy provides a solid foundation for solving this problem. Due to the fact that, the Differential Privacy Stochastic Gradient Descent (DPSDG) algorithm, as the standard gradient clipping method, often sacrifices the high performance of the model to improve the dynamic adaptability of the sample features. We propose a differential privacy adaptive gradient clipping method based on the diagonal Hessian Matrix, called DPHSGD, which dynamically adjusts the clipping threshold using the second-order derivative information of the loss function, balancing privacy protection and model utility. The experimental results show that this method improves by about 1% to 3% compared to benchmark methods in terms of classification accuracy on the MNIST dataset. At the same time, in order to cope with the extra time cost caused by calculating the diagonal Hessian matrix, a dynamic interval Hessian update mechanism is proposed, which effectively reduces calculation cost and improves the training efficiency.