Demystifying the Impact of Open-Source Machine Learning Libraries on Software Analytics

Machine learning (ML) classification techniques from various libraries have been widely introduced into software engineering (SE) to mine instructive insights, which help developers guarantee software quality. However, researchers would report instructive insights with no clear stability and consensus due to the indiscriminate use of various ML libraries. Such a lack of directive on using various ML libraries prevents developers from effectively applying instructive insights in practice. Therefore, through a case study of 23 popular software datasets across three task domains (i.e., software defect prediction, issue lifetime estimation, and code smell detection) in SE, in this article, we systematically study the impact of open-source ML libraries on performance consistency, performance stability, and model interpretation of six commonly used classifiers across two commonly used ML programming language (i.e., Python and R). We find that for a given classification technique: ML libraries with the tune setting cannot generate stable and consistent performance; ML libraries are sensitive in the interpretation of the model, even in case of the same parameter settings; and ML libraries from R would generate more stable and higher performance. Based on these findings, we suggest that future work in SE should indicate the specific ML libraries with the specific parameter settings that are used to discover the instructive insights for their tasks; try the ML libraries from R to build the models with high and stable performance for the software tasks; and use the same ML libraries that are used to select important features to construct the classification model.