On the Use of Hyperparameter Optimization in Big Data Processing Pipelines: A Case Study

The term "Data Analytics" sparks a wide range of multidisciplinary fields since it requires a high analytical expertise in different domains. Data Analytics applications include many successive steps, such as data collection, outlier detection, missing data imputation, feature selection, clustering analysis, classification model selection and result interpretation. The whole procedure can be seen as a chain of steps, organized in a pipeline manner, where the output of the upper layer is the input of the lower layer. Thus, the tasks of every step depend directly on the results of the previous steps. Moreover, there are alternative algorithmic methods for each step in Data Analytics. Opting for one methodology over the other requires high-skilled data scientists with a huge technical background. The key point of such a decision is optimizing the parameters of every layer in the pipeline. In this study, we develop an automated pipeline with different layers. Every layer contains several methods. We investigate the implementation of a suitable hyperparameter optimization algorithm, which allows the pipeline to be autonomous and select wisely the best algorithm for every layer. We discuss the specifics of the proposed prototype and the details of the used frameworks. We evaluate the prototype with an experimental study. The results are pertinent.