Enhancing Machine Learning Models Using a Hierarchical Two-Level Genetic Algorithm for Feature Engineering

Deep learning models are widely used in data-driven applications due to their high predictive performance, but their lack of interpretability limits their applicability in domains requiring transparency. In contrast, classical machine learning models offer interpretability but often at the cost of lower accuracy. One way to boost the performance of simpler machine learning models is to use feature engineering. However, as feature engineering requires evaluating multiple combinations of features to select the more relevant ones, this task is not trivial. This study addresses this issue by introducing a hierarchical genetic algorithm-based approach. Our approach uses the Non-Dominated Sorting Genetic Algorithm II (NSGA-II) to balance between classification accuracy and feature set size. In the first level of this approach, we used bootstrapping, feature transformations, and ensemble methods to derive a set of optimal features. In the second level, two different strategies were used to combine these candidate sets to find the final set. We assessed our approach on twelve datasets, encompassing different fields, achieving an average F1 score increment of 1.5%, while reducing the feature set size by 54.5%. Compared to traditional filter-based selection methods, our approach demonstrated competitive or superior performance. Our approach is implemented in an open-source Python library, feature-gen, providing a practical tool for enhancing the performance and interpretability of machine learning models. Overall, our approach underscores the value of evolutionary algorithms and ensemble learning in enhancing the performance of interpretable machine learning models.