Fast and robust selection of highly-correlated features in regression problems

Feature selection for regression problems can be highly beneficial in terms of robustness and execution speed. The Correlation-based Feature Selection (CFS) algorithm, in the attempt to find the best feature subset, evaluates different subsets and selects the one with the highest “goodness”. Such goodness is based on the co-relation between the addition of all features in the subset with the output variable. However, such a simple addition assumes that all features have the same weight in the output variable. This, in turn, assumes that all features are uncorrelated with each other. By considering an optimal weighting instead, a more robust measurement of the goodness can be obtained; however, such weighting is computationally expensive. In this paper, a feature selection algorithm named “R-fast” which considers the β coefficients in the standardized linear regression model as optimal weights is proposed. We also present a technique to quickly estimate approximations to the β coefficients. Our algorithm was evaluated using Multiple Regression Analysis (MRA) over 8 synthetic and 10 real-world datasets. Results show that, when selecting features with our proposed algorithm, MRA's performance is better than or equal to those obtained when selecting features with others well-known filter algorithms and when no selection is performed.