Preconditioned Gradient Descent for Sketched Mixture Learning

Sketching consists of reducing the dimensionality of data samples, generally by summarizing their essential features, such as retaining a small number of empirical moments. The reduced representation is called a sketch. In this paper, a Preconditioned Gradient Descent algorithm (PGD) is proposed to estimate the parameter of mixture models (MM) in arbitrary dimensions by minimizing the non-convex quadratic loss between the sketch and the characteristic function of an MM of varying parameters. Preconditioning is introduced to dynamically adapt the descent direction to the local landscape of the objective function, accelerating convergence, with no computational overhead per iteration compared with vanilla GD. An analysis of the linear convergence rate of PGD is conducted, and numerical simulations showcase the method's effectiveness, particularly when the weight of the classes is unbalanced or when a substantial number of data samples is available.