A Novel Feature Filtering Mechanism to Avoid Concept Shifts from Deteriorating Model Predictability

Traditional filter-based feature selection methods, which identify informative features based on their correlation with labels or large variances, often fail to address concept shifts-time-varying changes that diminish a machine learning model’s predictive accuracy when training patterns do not hold for the test set. To tackle this issue, we introduce a novel technique that eliminates time-varying features to enhance model robustness. We start by dividing the data into chronologically ordered training and validation sets, analyzing each feature’s behavior over time to capture its evolving patterns. We apply the Kolmogorov-Smirnov (KS) test to identify and remove features with significant distributional changes between the training and validation periods. Unlike previous studies that use the KS test for feature selection based on statistical properties, our approach leverages the KS test to specifically target time-varying features that lead to dataset drift. By filtering out these features, we prevent the model from learning outdated patterns, thereby improving the quality of the training data and enhancing the model’s predictive accuracy. Furthermore, this strategy reduces the model’s ability to distinguish between training and testing samples in discriminative learning, indicating its effectiveness in addressing dataset drift. Unlike traditional methods that split datasets primarily for hyperparameter tuning or early stopping, our approach directly assesses the chronological consistency of feature distributions to mitigate concept shifts. We validate this method through real-world experiments in fund transfer fraud detection and financial market investments—domains frequently affected by such shifts. The results demonstrate significant improvements in model predictability, highlighting the value of removing time-varying features in enhancing machine learning performance.