A Note on Arithmetic–Geometric Mean Inequality for Well-Conditioned Matrices

The generalization of the AM-GM inequality for matrices helps in understanding the difference between with-replacement sampling and without-replacement sampling in stochastic gradient descent (SGD). However, it was previously shown that the matrix form of the AM-GM inequality does not hold for sets of five or more matrices [1], [2]. An open problem presented in COLT 2021 [3] suggested that this limitation may not apply to matrices with small condition numbers, proposing this as a conjecture. In this paper, we establish a weak matrix AM-GM inequality for positive-semidefinite matrices with small condition numbers. Our findings indicate that the performance of SGD without replacement can not be significantly worse than with-replacement sampling, while the converse does not hold.