Improving Classification Performance by Merging Distinct Feature Sets of Similar Quality Generated by Multiple Initializations of mRMR

The success of machine learning algorithms often depends on the combination of model size, computational cost and interpretability. One way to optimize these properties is feature selection. Computational cost and model size can be reduced by discarding features with low relevance. Furthermore, feature selection can provide a deeper understanding of the feature's importance. This work focuses on the minimal-redundancy-maximal-relevance algorithm (mRMR) which is a filter-method for feature selection that uses pair wise mutual information as a measure to decide which feature is relevant. The algorithm is initialized with the feature with the highest relevance according to the measure and an iterative algorithm selects the next feature which optimizes for a high relevance while maintaining a low redundancy to the previously selected features. This work extensively studies distinct feature sets which can be generated when running the mRMR algorithm multiple times using features of descending relevance as initialization. By exploiting information about the order in which the iterative algorithm chooses the features in the various runs, a strategy is proposed to generate a new combined feature set from all initializations. Applying the proposed strategy to four datasets of different sizes and two classification algorithms shows that the resulting feature sets are significantly better compared to the original mRMR algorithm for the given classification task. The proposed method is well-suited for cases where it is not feasible to use wrapper-methods to increase classification accuracy.