Gaussian Process as Feature Selection Method for Regression tasks: Comparison with Classic Methods

Feature selection is an essential phase in the creation of machine learning models, particularly for regression problems where irrelevant or redundant data may undermine the model’s interpretability and precision. This paper examines and contrasts the effectiveness of several feature selection methods, emphasizing an innovative application of Gaussian Processes (GPs), often employed for hyperparameter optimization, in feature selection. This contrasts with conventional feature selection algorithms that have been widely employed in regression modeling, including Principal Component Analysis (PCA), Least Absolute Shrinkage and Selection Operator (LASSO), and Recursive Feature Elimination (RFE). The primary research focus is: The efficacy of different feature selection procedures for regression problems, particularly in comparing Gaussian Processes with traditional methods. We evaluate the performance of each method for model intelligibility, computational efficiency, prediction accuracy, and feature importance across several regression datasets. Furthermore, we examine the significance of feature selection for enhancing model performance in high-dimensional, noisy, or irrelevant datasets, as well as for ensuring resilience in generalization. Through various tests, we demonstrate that while traditional methods like RFE and LASSO are effective in certain contexts, Gaussian Processes offer a flexible, probabilistic approach that can exceed current techniques in achieving an optimal balance between model complexity and performance.