Statistical Similarity Measure for Interpreting Important Features in Machine Learning

Machine learning(ML)-based feature selection algorithms use specific techniques to assess feature relevance. In medical datasets, many attributes can be irrelevant for classification. Many features in these datasets are often irrelevant for classification tasks, and removing them can improve efficiency without sacrificing accuracy. The study uses techniques like Random Forest and Permutation Feature Importance to determine significant features from datasets provided by the University of California, Irvine (UCI). To assess feature relevance, statistical tests such as the Kolmogorov-Smirnov statistic and t-test are employed, along with the Mahalanobis Distance Measure to filter important features. The research highlights the value of feature selection in improving model performance. The dataset used demonstrated a 96% accuracy, 97% F1 score, and 98% AUC in 5-fold cross-validation with full features. When reduced to 50% of the most relevant features, the performance improved with an AUC of 99%, indicating better results with fewer features. The findings suggest that statistical similarity metrics like the Mahalanobis Distance Measure can be effective for feature selection, improving model efficiency and accuracy.