MP-Boost: Minipatch Boosting via Adaptive Feature and Observation Sampling

Boosting methods are among the best generalpurpose and off-the-shelf machine learning approaches, gaining widespread popularity. In this paper, we seek to develop a boosting method that yields comparable accuracy to popular AdaBoost and gradient boosting methods, yet is faster computationally and whose solution is more interpretable. We achieve this by developing MP-Boost, an algorithm loosely based on AdaBoost that learns by adaptively selecting small subsets of instances and features, or what we term minipatches (MP), at each iteration. By sequentially learning on tiny subsets of the data, our approach is computationally faster than classic boosting algorithms. MP-Boost upweights important features and challenging instances, hence adaptively selects the most relevant minipatches for learning. The learned probability distributions aid in interpretation of our method. We empirically demonstrate the interpretability and comparative accuracy of our algorithm on a variety of binary classification tasks.