S-ADMM: Optimizing Machine Learning on Imbalanced Datasets Based on ADMM

Nowadays, addressing large-scale machine learning problems using high-performance computing (HPC) clusters has gained significant importance. The alternating direction method of multipliers (ADMM) is widely used in machine learning for solving optimization problems on clusters. However, ADMM's performance is greatly affected by imbalanced datasets on the HPC cluster. In this paper, we propose the distributed shunt ADMM with a new adaptive penalty method based on the hybrid MPI/OpenMP programming model (S-ADMM). The proposed shunt strategy chooses different sub-problem optimization algorithms to improve the accuracy with the imbalanced datasets. Additionally, we design a novel adaptive penalty parameter method and improve a sub-problem optimization algorithm for S-ADMM. The adaptive penalty parameter method accelerates the algorithm's convergence and the sub-problem optimization algorithm improves the training efficiency of ADMM. Moreover, S-ADMM reduces communication cost by exchanging parameters among nodes using the MPI and saves calculation time by parallel computation within nodes via OpenMP threads. For the SVM classification problem, experiments conducted on the Tianhe-2 supercomputing platform show that S-ADMM has competitive running efficiency and up to 43% accuracy improvement compared to existing distributed ADMM implemented with pure MPI or MPI/OpenMP on imbalanced datasets.