Dynamic Instance-wise Joint Feature Selection and Classification

In this article, a dynamic instance-wise joint feature selection and classification framework during testing is presented. Specifically, the proposed framework sequentially selects features one at a time for each data instance, given previously selected features, and stops this process to classify the instance once it determines that including additional features will not improve the final classification decision. In contrast to most of the existing work that utilizes a set of features, common for all data instances, the proposed framework utilizes different features to classify each data instance. An optimization problem is defined for each data instance in terms of the number of selected features and the associated classification accuracy. The optimum solution is derived, and its structure is analyzed. Based on the optimum solution and its properties, two new algorithms are designed. The expected number of features needed to achieve a given classification accuracy is also analytically derived. Finally, the performance of the proposed algorithms is illustrated on 11 public datasets, thus demonstrating their effectiveness and scalability across a broad range of application domains.

Impact Statement—In many domains, including but not limited to medicine and criminal justice, experts need to reach an accurate decision in a timely manner using limited resources (e.g., costly tests and time-consuming evidence collection). At the same time, it is desirable to tailor decisions to each individual case (e.g., patient and defendant). Most of the existing machine learning algorithms, however, ignore resource constraints and/or acquire a general solution for all cases, while not scaling in big data settings. The algorithms proposed in this article address such challenges enabling tailored and timely decision making. With a reduction of up to 66% in the average number of features, while maintaining similar accuracy levels, the proposed algorithms can be used for dynamic instance-wise joint feature selection and classification in scenarios involving over one million variables.