Optimal Feature Selection for Dimensionality Reduction with Preserved Predictive Performance in High-Dimensional Data

High-dimensional data tend to cause issues like higher computational expense, overfitting, and loss of model interpretability. This paper introduces an optimal feature selection technique with focus on dimensionality reduction and predictive performance preservation under a wide variety of machine learning algorithms. With benchmark datasets like UCI Human Activity Recognition (561 features) and Gene Expression Cancer RNA-Seq (20,531 features), the approach was successful with dimension reduction of up to 95% with less than 2% loss in classification performance. Relative performance against classical methods like PCA and mutual information yielded a 12-18% gain on F1-score and 25% gain on training time. The findings illustrate how the method preserves model scalability and stability, thus applied in high-dimensional applications like sensor measurements and bioinformatics. The article highlights the importance of the critical dimensionality vs. predictability integrity trade-off.