Impact of Hyperparameter Tuning in Classifying Highly Imbalanced Big Data

Working with Machine Learning algorithms and Big Data, one may be tempted to skip the process of hyperparameter tuning, since algorithms generally take longer to train on larger datasets. Hyperparameter tuning is not something we get for free; one must allocate either more computing time or resources to run more iterations of experiments with different hyperparameter settings to find their optimal values. For small datasets the extra resources needed may be negligible, but for larger datasets resources necessary for tuning may be considerable. In this study, due to the size of the data we use, we find experiments where we do hyperparameter tuning take many times longer to complete than experiments where we do not do any tuning. Here, our contribution is to provide evidence that the investment in computing resources for hyperparameter tuning pays off, since we show that, for experiments involving highly imbalanced Big Data, we obtain better results when we incorporate hyperparameter tuning. With regard to the performance of CatBoost and LightGBM classifiers, we compare both default and tuned hyperparameter values. Furthermore our experiments encompass different techniques for encoding high-cardinality categorical features. We find in all cases, regardless of the classifier or encoding technique for categorical features, classifiers with tuned values for hyperparameters yield better results than those with default values. To the best of our knowledge, we are the first to do such a study on hyperparameter tuning to analyze the performance of LightGBM and CatBoost in classifying highly imbalanced Big Data.