Binary linear classification and feature selection via generalized approximate message passing

For the problem of binary linear classification and feature selection, we propose algorithmic approaches to classifier design based on the generalized approximate message passing (GAMP) algorithm, recently proposed in the context of compressive sensing. Our work focuses on the regime where the number of features greatly exceeds the number of training examples, but where only a few features suffice for accurate classification. We show that sum-product GAMP can be used to (approximately) minimize the classification error rate and max-sum GAMP can be used to minimize a wide variety of regularized loss functions. Furthermore, we describe an expectation-maximization (EM)-based scheme to learn the associated model parameters online, as an alternative to cross-validation, and we show that GAMPs state evolution framework can be used to accurately predict the misclassification rate. Finally, we present a brief numerical study to confirm the efficacy, flexibility, and speed afforded by our GAMP-based approaches to binary classification.