Accelerating Join of Distributed Datasets by a Given Criterion

This article discusses the operation of joining distributed datasets by a given criterion in distributed systems. A critical analysis of literature data on the architecture of distributed data warehouses and typical methods for joining datasets was carried out, limiting stages that slow down the process were identified. A method for accelerating the operation of data joining according to a given criterion is proposed, on the basis of which an algorithm is developed and implemented in the Apache Spark data processing environment. Experimental studies confirming the efficiency of the developed method were performed. The results of the experiments show that the proposed method can significantly increase speed of the operation compared to existing solutions. From the presented experimental data, it can be seen that for 2 TB data, the algorithm made it possible to perform the merge operation ~ 37% faster than the standard algorithm offered by the Spark SQL library, for 7 TB data it was already ~ 47%.