Analysis of dual-stage filtration and validation of high-dimensional real process data for creation of machine learning algorithms

The article presents an analysis of the dual-stage filtration and validation of high-dimensional real process data required for the creation of a machine learning algorithm. The dual-stage filtration analysis has been carried out by means of a special type of artificial neural network clustering of high-dimensional data, called the self-organizing maps. By means of creating self-organizing maps, we can identify the data that do not belong to a specific group, and which is eliminated from the group, in order to obtain filtered high-dimensional real process data. If filtering is repeated on previously filtered high-dimensional real process data, double-filtered data are obtained. And double-filtered high-dimensional real process data is used in the machine learning process to create an algorithm. The quality of the created algorithm depends also on the data used and is checked in the validation procedure. The validation procedure is aimed at determining errors in the results delivered by the algorithm and real process data. We have validated various algorithm structures, with non-filtered, once-filtered, and double-filtered high-dimensional real process data, and compared the results. The results of the analysis show that the quality of the created machine learning algorithm improves with the dual-stage filtration and such an algorithm delivers better results or results with fewer errors.