CPP: Compensated Post-Training Pruning Approach for On-Device Large Language Model Services

Using pruning techniques to prune redundant weights in large language models (LLMs) for model size reduction, enabling deployment on devices to deliver high-quality services, has garnered significant attention from industry and academia. However, most existing pruning methods suffer from two major problems: (1) they rely on operations with high computational complexity, such as Hessian matrix calculation, and (2) high pruning rate leads to a significant decrease in model accuracy. To this end, this paper proposes a low resource requirement and low accuracy loss post-training pruning approach, namely the compensated post-training pruning method (CPP) for on-device LLM services. First, CPP employs singular value decomposition on weight matrices, sorting the decomposed singular values in descending order. Based on the pruning ratio, it retains the principal eigenvector corresponding to the larger singular values. To maintain consistent output feature distributions, CPP applies orthogonal transformations to input data and weight matrices, leveraging the principle of matrix orthogonality invariance. Compared with pruning approaches based on the Hessian matrix, CPP does not need to iteratively calculate the second derivative, thereby avoiding a high computational overhead. Second, the CPP incorporates bias compensation to use valuable information in pruned weights to improve the accuracy of the model. It constructs mapping relationships between input features and pruned weights through tensor decomposition techniques to generate bias compensation terms. These terms fine-tune the output of each layer, reducing pruning-induced errors from ratio-based pruning, and consequently improving model accuracy. Finally, CPP is validated through comprehensive experiments on nine benchmark datasets (e.g., WikiText-2 and PIQA) using large language models (i.e., LLaMA and OPT). Specifically, compared with the FLAP, CPP demonstrates 6.68% reduction in average pruning time. At pruning ratios of 10% and 20%, CPP achieves 2. 14% and 1.45% reductions in perplexity, respectively, along with 1.22% to 3.43% improvement in zeroshot inference accuracy.